Intelligent agents are showing increasing promise for clinical decision-making in a variety of healthcare settings. While a substantial body of work has contributed to the best strategies to convey these agents’ decisions to clinicians, few have considered the impact of personalizing and customizing these communications on the clinicians’ performance and receptiveness. This raises the question of how intelligent agents should adapt their tone in accordance with their target audience. We designed two approaches to communicate the decisions of an intelligent agent for breast cancer diagnosis with different tones: a suggestive (non-assertive) tone and an imposing (assertive) one. We used an intelligent agent to inform about: (1) number of detected findings; (2) cancer severity on each breast and per medical imaging modality; (3) visual scale representing severity estimates; (4) the sensitivity and specificity of the agent; and (5) clinical arguments of the patient, such as pathological co-variables. Our results demonstrate that assertiveness plays an important role in how this communication is perceived and its benefits. We show that personalizing assertiveness according to the professional experience of each clinician can reduce medical errors and increase satisfaction, bringing a novel perspective to the design of adaptive communication between intelligent agents and clinicians.

In this paper, we developed BreastScreening-AI within two scenarios for the classification of multimodal beast images: (1) Clinician-Only; and (2) Clinician-AI. The novelty relies on the introduction of a deep learning method into a real clinical workflow for medical imaging diagnosis. We attempt to address three high-level goals in the two above scenarios. Concretely, how clinicians: i) accept and interact with these systems, revealing whether are explanations and functionalities required; ii) are receptive to the introduction of AI-assisted systems, by providing benefits from mitigating the clinical error; and iii) are affected by the AI assistance. We conduct an extensive evaluation embracing the following experimental stages: (a) patient selection with different severities, (b) qualitative and quantitative analysis for the chosen patients under the two different scenarios. We address the high-level goals through a real-world case study of 45 clinicians from nine institutions. We compare the diagnostic and observe the superiority of the Clinician-AI scenario, as we obtained a decrease of 27% for False-Positives and 4% for False-Negatives. Through an extensive experimental study, we conclude that the proposed design techniques positively impact the expectations and perceptive satisfaction of 91% clinicians, while decreasing the time-to-diagnose by 3 min per patient.

Exploiting well-labeled training sets has led deep learning models to astonishing results for counting biological structures in microscopy images. However, dealing with weak multi-rater annotations, i.e., when multiple human raters disagree due to non-trivial patterns, remains a relatively unexplored problem. More reliable labels can be obtained by aggregating and averaging the decisions given by several raters to the same data. Still, the scale of the counting task and the limited budget for labeling prohibit this. As a result, making the most with small quantities of multi-rater data is crucial. To this end, we propose a two-stage counting strategy in a weakly labeled data scenario. First, we detect and count the biological structures; then, in the second step, we refine the predictions, increasing the correlation between the scores assigned to the samples and the raters' agreement on the annotations. We assess our methodology on a novel dataset comprising fluorescence microscopy images of mice brains containing extracellular matrix aggregates named perineuronal nets. We demonstrate that we significantly enhance counting performance, improving confidence calibration by taking advantage of the redundant information characterizing the small sets of available multi-rater data.

Deep neural networks have been tremendously successful at segmenting objects in images. However, it has been shown they still have limitations on challenging problems such as the segmentation of medical images. The main reason behind this lower success resides in the reduced size of the object in the image. In this paper we overcome this limitation through a cyclic collaborative framework, CyCoSeg. The proposed framework is based on a deep active shape model (D-ASM), which provides prior information about the shape of the object, and a semantic segmentation network (SSN). These two models collaborate to reach the desired segmentation by influencing each other: SSN helps D-ASM identify relevant keypoints in the image through an Expectation Maximization formulation, while D-ASM provides a segmentation proposal that guides the SSN. This cycle is repeated until both models converge. Extensive experimental evaluation shows CyCoSeg boosts the performance of the baseline models, including several popular SSNs, while avoiding major architectural modifications. The effectiveness of our method is demonstrated on the left ventricle segmentation on two benchmark datasets, where our approach achieves one of the most competitive results in segmentation accuracy. Furthermore, its generalization is demonstrated for lungs and kidneys segmentation in CT scans.

Video analysis often requires locating and tracking target objects. In some applications, the localization system has access to the full video, which allows fine-grain motion information to be estimated. This paper proposes capturing this information through motion fields and using it to improve the localization results. The learned motion fields act as a model-agnostic temporal regularizer that can be used with any localization system based on keypoints. Unlike optical flow-based strategies, our motion fields are estimated from the model domain, based on the trajectories described by the object keypoints. Therefore, they are not affected by poor imaging conditions. The benefits of the proposed strategy are shown on three applications: 1) segmentation of cardiac magnetic resonance; 2) facial model alignment; and 3) vehicle tracking. In each case, combining popular localization methods with the proposed regularizer leads to improvement in overall accuracies and reduces gross errors.

The second edition of the International Workshop on Multimodal Conversational AI puts forward a diverse set of contributions that aim to brainstorm this new field. Conversational agents are now becoming a commodity as this technology is being applied to a wide range of domains. Healthcare, assisting technologies, e-commerce, information seeking, are some of the domains where multimodal conversational AI is being explored. The wide use of multimodal conversational agents exposes the many challenges in achieving more natural, human-like, and engaging conversational agents. The research contributions of the Workshop actively address several of relevant challenges: How to include assistive-technologies in dialog systems? How can agents engage in negotiation in dialogs? How to handle the embodiment of conversational agents?

Object recognition and localization is still a very challenging problem, despite recent advances in deep learning (DL) approaches, especially for objects with varying shapes and appearances. Statistical models, such as an Active Shape Model (ASM), rely on a parametric model of the object, allowing an easy incorporation of prior knowledge about shape and appearance in a principled way. To take advantage of these benefits, this paper proposes a new ASM framework that addresses two tasks: (i) comparing the performance of several image features used to extract observations from an input image; and (ii) improving the performance of the model fitting by relying on a probabilistic framework that allows the use of multiple observations and is robust to the presence of outliers. The goal in (i) is to maximize the quality of the observations by exploring a wide set of handcrafted features (HOG, SIFT, and texture templates) and more recent DL-based features. Regarding (ii), we use the Generalized Expectation-Maximization algorithm to deal with outliers and to extend the fitting process to multiple observations. The proposed framework is evaluated in the context of facial landmark fitting and the segmentation of the endocardium of the left ventricle in cardiac magnetic resonance volumes. We experimentally observe that the proposed approach is robust not only to outliers, but also to adverse initialization conditions and to large search regions (from where the observations are extracted from the image). Furthermore, the results of the proposed combination of the ASM with DL-based features are competitive with more recent DL approaches (e.g. FCN [1], U-Net [2] and CNN Cascade [3]), showing that it is possible to combine the benefits of statistical models and DL into a new deep ASM probabilistic framework.

Finding good representations for videos is becoming increasingly more important to enable an efficient analysis and comparison, with potential applications in sports, surveillance, news, or web services. This paper proposes a new representation of videos based on human pose. Rather than looking at conventional features, our method relies only on human pose detections to characterize the video. This approach provides a powerful tool for the efficient analysis of videos of human activities, particularly for video summarization and retrieval. We evaluate the proposed representation on the following tasks: 1) computing video statistics, such as the main poses and viewpoint preferences; 2) partitioning videos into a collection of short clips that will compose the video summary; and 3) retrieving frames or scenes with specific poses from videos. Results show that the proposed approach is able to successfully perform these tasks.

Obtaining the segmentation of an object in a sequence of images is usually achieved using a tracking methodology. However, in some applications, the whole sequence is available beforehand. This means that the segmentations can be determined simultaneously for all the frames in the sequence and taking into account the motion of the object. This paper proposes a new framework to incorporate motion information in the segmentation of image sequences using an active shape model (ASM). The motion of the object is modeled using a vector field, which is learned and refined online as the segmentation algorithm proceeds. The vector field is determined from the trajectories described by ASM points throughout the sequence. The vector field, in turn, influences the estimation of the ASM parameters by acting as a regularizer, ensuring that the segmentations are in agreement with the expected motion. The results show that coupling these models during the segmentation leads to an increase in performance, in particular by guarantee more consistent segmentations and by avoid gross errors in more challenging frames.

This paper proposes a new approach for the segmentation of the endocardium of the left ventricle using short axis magnetic resonance (MR) images. The proposed method comprises two main stages. First, each image is converted to polar coordinates, and an edge map is computed from the transformed image. Then, the contour of the left ventricle (LV) is estimated by computing the optimal path along the edge map, using a dynamic programming approach. The system is evaluated on a public database comprising 660 magnetic resonance volumes and the results testify its usefulness both in terms of running time and accuracy. The proposed methodology is able to segment a whole volume in 1.5 seconds achieving an average Dice similarity coefficient of 85.9% (8.3%), which compares favorably with related state-of-the-art methods.

The 3D segmentation of the left ventricle (LV) in cardiac MRI is a challenging problem, due to the presence of other anatomical structures and artifacts (outliers) around the LV. In this paper, a new formulation of a Robust Active Shape Model (RASM) is presented that is able to deal with those outliers. Instead of using the traditional one-to-one mapping of edge points and model points to compute the shape model parameters, the proposed approach uses a one-to-many mapping strategy and groups these edge points into edge segments (strokes). Then, a probabilistic framework provides a robust estimation of the model parameters, in which the influence in the segmentation of the unreliable outliers is reduced. The proposed method was tested on a public dataset comprising 660 volumes. The results indicate that this methodology provides accurate segmentations that are competitive with other state-of-the art methods.

Statistical shape models have been extensively used in a wide range of applications due to their effectiveness in providing prior shape information for object segmentation problems. The most popular method is the Active Shape Model (ASM). However, accurately fitting the shape model to an object boundary under a cluttered environment is a challenging task. Under such assumptions, the model is often attracted towards invalid observations (outliers), leading to meaningless estimates of the object boundary. In this paper, we propose a novel algorithm that improves the robustness of ASM in the presence of outliers. The proposed framework assumes that both type of observations (valid observations and outliers) are detected in the image. A new strategy is devised for treating the data in different ways, depending on the observations being considered as valid or invalid. The proposed algorithm assigns a different weight to each observation. The shape parameters are recursively updated using the Expectation-Maximization method, allowing a correct and robust fit of the shape model to the object boundary in the image. Two estimation criteria are considered: 1) the maximum likelihood criterion; and 2) the maximum a posteriori criterion that uses priors for the unknown parameters. The methods are tested with synthetic and real images, comprising medical images of the heart and image sequences of the lips. The results are promising and show that this approach is robust in the presence of outliers, leading to a significant improvement over the standard ASM and other state of the art methods.

The segmentation of the left ventricle (LV) is an important task to assess the cardiac function in ultrasound images of the heart. This paper presents a novel methodology for the segmentation of the LV in three-dimensional (3-D) echocardiographic images based on the probabilistic data association filter (PDAF). The proposed methodology begins by initializing a 3-D deformable model either semiautomatically, with user input, or automatically, and it comprises the following feature hierarchical approach: 1) edge detection in the vicinity of the surface (low-level features); 2) edge grouping to obtain potential LV surface patches (mid-level features); and 3) patch filtering using a shape-PDAF framework (high-level features). This method provides good performance accuracy in 20 echocardiographic volumes, and compares favorably with the state-of-the-art segmentation methodologies proposed in the recent literature.

Active shape models (ASM) have been extensively used in object segmentation problems because they constrain the solution, using shape statistics. However, accurately fitting an ASM to an image prone to outliers is difficult and poor results are often obtained. To overcome this difficulty we propose a robust algorithm based on the Expectation-Maximization framework that assigns different weights (confidence degrees) to the observations extracted from the image. This reduces the influence of outliers since they often receive low weights. We tested the proposed algorithm with synthetic and real images (e.g., lip images and cardiac ultrasound images) achieving promising results. The proposed algorithm performs significantly better than the standard ASM implementation.

The segmentation of the left ventricle (LV) is an important tool to assess the cardiac function in ultrasound images of the heart. This paper presents a methodology for the segmentation of the LV in 3D echocardiography that is based on the probabilistic data association filter (PDAF). The proposed methodology comprises the following feature hierarchical approach: (i) edge detection in the vicinity of the surface (low level features); (ii) edge grouping to obtain potential LV surface patches (mid-level features); and (iii) patch filtering using a shape-PDAF framework (high level features). Also, we propose an automatic procedure to initialize the 3D deformable model. We show that the proposed methodology achieves remarkable accuracy between the obtained contour and expertise medical ground truth and compares favorably with the state-of-the-art segmentation methodologies.