We present an accurate, fast and efficient method for segmentation and muscle mask propagation in 3D freehand ultrasound data, towards accurate volume quantification. A deep Siamese 3D Encoder-Decoder network that captures the evolution of the muscle appearance and shape for contiguous slices is deployed. We use it to propagate a reference mask annotated by a clinical expert. To handle longer changes of the muscle shape over the entire volume and to provide an accurate propagation, we devise a Bidirectional Long Short Term Memory module. Also, to train our model with a minimal amount of training samples, we propose a strategy combining learning from few annotated 2D ultrasound slices with sequential pseudo-labeling of the unannotated slices. We introduce a decremental update of the objective function to guide the model convergence in the absence of large amounts of annotated data. After training with a few volumes, the decremental update strategy switches from a weak supervised training to a few-shot setting. Finally, to handle the class-imbalance between foreground and background muscle pixels, we propose a parametric Tversky loss function that learns to penalize adaptively the false positives and the false negatives. We validate our approach for the segmentation, label propagation, and volume computation of the three low-limb muscles on a dataset of 61600 images from 44 subjects. We achieve a Dice score coefficient of over 95% and a volumetric error of 1.6035 ± 0.587%.

Early breast cancer screening through mammography produces every year millions of images worldwide. Despite the volume of the data generated, these images are not systematically associated with standardized labels. Current protocols encourage giving a malignancy probability to each studied breast but do not require the explicit and burdensome annotation of the affected regions. In this work, we address the problem of abnormality detection in the context of such weakly annotated datasets. We combine domain knowledge about the pathology and clinically available image-wise labels to propose a mixed self- and weakly supervised learning framework for abnormalities reconstruction. We also introduce an auxiliary classification task based on the reconstructed regions to improve explainability. We work with high-resolution imaging that enables our network to capture different findings, including masses, micro-calcifications, distortions, and asymmetries, unlike most state-of-the-art works that mainly focus on masses. We use the popular INBreast dataset as well as our private multi-manufacturer dataset for validation and we challenge our method in segmentation, detection, and classification versus multiple state-of-the-art methods. Our results include image-wise AUC up to 0.86, overall region detection true positives rate of 0.93, and the pixel-wise${F}_{{1}}$score of 64% on malignant masses.

In a low-statistics PET imaging context, the positive bias in regions of low activity is a burning issue. To overcome this problem, algorithms without the built-in non-negativity constraint may be used. They allow negative voxels in the image to reduce, or even to cancel the bias. However, such algorithms increase the variance and are difficult to interpret since the resulting images contain negative activities, which do not hold a physical meaning when dealing with radioactive concentration. In this paper, a post-processing approach is proposed to remove these negative values while preserving the local mean activities. Its original idea is to transfer the value of each voxel with negative activity to its direct neighbors under the constraint of preserving the local means of the image. In that respect, the proposed approach is formalized as a linear programming problem with a specific symmetric structure, which makes it solvable in a very efficient way by a dual-simplex-like iterative algorithm. The relevance of the proposed approach is discussed on simulated and on experimental data. Acquired data from an yttrium-90 phantom show that on images produced by a non-constrained algorithm, a much lower variance in the cold area is obtained after the post-processing step, at the price of a slightly increased bias. More specifically, when compared with the classical OSEM algorithm, images are improved, both in terms of bias and of variance.

We address the problem of multimodal image registration using a supervised learning approach. We pose the problem as a regression task, whose goal is to estimate the unknown geometric transformation from the joint appearance of the fixed and moving images. Our method is based on (i) context-aware features, which allow us to guide the registration using not only local, but also global structural information, and (ii) regression forests to map the very large contextual feature space to transformation parameters. Our approach improves the capture range, as we demonstrate on the publicly available IXI dataset. Furthermore, it can also handle difficult settings where other similarity metrics tend to fail; for instance, we show results on the deformable registration of Intravascular Ultrasound (IVUS) and Histology images. These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Multimodal registration is a challenging problem due the high variability of tissue appearance under different imaging modalities. The crucial component here is the choice of the right similarity measure. We make a step towards a general learning-based solution that can be adapted to specific situations and present a metric based on a convolutional neural network. Our network can be trained from scratch even from a few aligned image pairs. The metric is validated on intersubject deformable registration on a dataset different from the one used for training, demonstrating good generalization. In this task, we outperform mutual information by a significant margin.

This paper describes a novel method to acquire depth images using a pair of ToF (Time-of-Flight) cameras. As opposed to approaches that filter, calibrate or do 3D reconstructions posterior to the image acquisition, we combine the measurements of the two cameras within a modified acquisition procedure. The new proposed stereo-ToF acquisition is composed of three stages during which we actively modify the infrared lighting of the scene: first, the two cameras emit an infrared signal one after the other (stages 1 and 2), and then, simultaneously (stage 3). Assuming the scene is static during the three stages, we gather the depth measurements obtained with both cameras and define a cost function to optimize the two depth images. A qualitative and quantitative evaluation of the performance of the proposed stereo-ToF acquisition is provided both for simulated and real ToF cameras. In both cases, the stereo-ToF acquisition produces more accurate depth measurements. Moreover, an extension to the multi-view ToF case and a detailed study on the interference specifications of the system are included.

Gastro-intestinal (GI) endoscopy is a widely used clinical procedure for screening and surveillance of digestive tract diseases ranging from Barrett's Oesophagus to oesophageal cancer. Current surveillance protocol consists of periodic endoscopic examinations performed in 3-4 month intervals including expert's visual assessment and biopsies taken from suspicious tissue regions. Recent development of a new imaging technology, called probe-based confocal laser endomicroscopy (pCLE), enabled the acquisition of in vivo optical biopsies without removing any tissue sample. Besides its several advantages, i.e., noninvasiveness, real-time and in vivo feedback, optical biopsies involve a new challenge for the endoscopic expert. Due to their noninvasive nature, optical biopsies do not leave any scar on the tissue and therefore recognition of the previous optical biopsy sites in surveillance endoscopy becomes very challenging. In this work, we introduce a clustering and classification framework to facilitate retargeting previous optical biopsy sites in surveillance upper GI-endoscopies. A new representation of endoscopic videos based on manifold learning, "endoscopic video manifolds" (EVMs), is proposed. The low dimensional EVM representation is adapted to facilitate two different clustering tasks; i.e., clustering of informative frames and patient specific endoscopic segments, only by changing the similarity measure. Each step of the proposed framework is validated on three in vivo patient datasets containing 1834, 3445, and 1546 frames, corresponding to endoscopic videos of 73.36, 137.80, and 61.84 s, respectively. Improvements achieved by the introduced EVM representation are demonstrated by quantitative analysis in comparison to the original image representation and principal component analysis. Final experiments evaluating the complete framework demonstrate the feasibility of the proposed method as a promising step for assisting the endoscopic expert in retargeting the optical biopsy sites.

In this paper, we present a method for human full-body pose estimation from Time-of-Flight (ToF) camera images. Our approach consists of robustly detecting anatomical landmarks in the 3D data and fitting a skeleton body model using constrained inverse kinematics. Instead of relying on appearance-based features for interest point detection that can vary strongly with illumination and pose changes, we build upon a graph-based representation of the ToF depth data that allows us to measure geodesic distances between body parts. As these distances do not change with body movement, we are able to localize anatomical landmarks independent of pose. For differentiation of body parts that occlude each other, we employ motion information, obtained from the optical flow between subsequent ToF intensity images. We provide a qualitative and quantitative evaluation of our pose tracking method on ToF sequences containing movements of varying complexity.

This paper describes a novel method to acquire depth images using a pair of ToF (Time of Flight) cameras. As opposed to approaches that filter, calibrate or do 3D reconstructions posterior to the image acquisition, we propose to combine the measurements of the two cameras at the acquisition level. To do so, we define a three-stages procedure, during which we actively modify the infrared lighting of the scene: first, the two cameras emit an infrared signal one after the other (stages 1 and 2), and then, simultaneously (stage 3). Assuming the scene is static during the three stages, we gather the depth measurements obtained with both cameras and define a cost function to optimize the two depth images. A quantitative evaluation of the performance of the proposed method for different objects and stereo configurations is provided based on a simulation of the ToF cameras. Results on real images are also presented. Both in simulation and real images the stereo-ToF acquisition produces more accurate depth measurements.

This paper describes an extension to the Monocular Simultaneous Localization and Mapping (MonoSLAM) method that relies on the images provided by a combined high resolution Time of Flight (HR-ToF) sensor. In its standard formulation MonoSLAM estimates the depth of each tracked feature as the camera moves. This depth estimation depends both on the quality of the feature tracking and the previous camera position estimates. Additionally, MonoSLAM requires a set of known features to initialize the scale of the map and the world coordinate system. We propose to use the combined high resolution ToF sensor to incorporate depth measures into the MonoSLAM framework while keeping the accuracy of the feature detection. In practice, we use a ToF (Time of Flight) and a high-resolution (HR) camera in a calibrated and synchronized set-up and modify the measurement model and observation updates of MonoSLAM. The proposed method does not require known features to initialize a map. Experiments show first, that the depth measurements in our method improve the results of camera localization when compared to the MonoSLAM approach using HR images alone; and second, that HR images are required for reliable tracking.

In this paper, we propose a method for detection and tracking of multiple planes in sequences of Time of Flight (ToF) depth images. Our approach extends the recent J-linkage algorithm for estimation of multiple model instances in noisy data to tracking. Instead of randomly selecting plane hypotheses in every image, we propagate plane hypotheses through the sequence of images, resulting in a significant reduction of computational load in every frame. We also introduce a multi-pass scheme that allows detecting and tracking planes of varying spatial extent along with their boundaries. Our qualitative and quantitative evaluation shows that the proposed method can robustly detect planes and consistently track the hypotheses through sequences of ToF images.

International audience

In this paper, an analysis of locally linear embedding (LLE) in the context of clustering is developed. As LLE conserves the local affine coordinates of points, shape protrusions as high-curvature regions of the surface are preserved. Also, LLEpsilas covariance constraint acts as a force stretching those protrusions and making them wider separated and lower dimensional. A novel scheme for unsupervised body-part segmentation along time sequences is thus proposed in which 3-D shapes are clustered after embedding. Clusters are propagated in time, and merged or split in an unsupervised fashion to accommodate changes of the body topology. Comparisons on synthetic, and real data with ground truth, are run with direct segmentation in 3-D by EM clustering and ISOMAP-based clustering. Robustness and the effects of topology transitions are discussed.

Matching articulated shapes represented by voxel-sets reduces to maximal sub-graph isomorphism when each set is described by a weighted graph. Spectral graph theory can be used to map these graphs onto lower dimensional spaces and match shapes by aligning their embeddings in virtue of their invariance to change of pose. Classical graph isomorphism schemes relying on the ordering of the eigenvalues to align the eigenspaces fail when handling large data-sets or noisy data. We derive a new formulation that finds the best alignment between two congruent K-dimensional sets of points by selecting the best subset of eigenfunctions of the Laplacian matrix. The selection is done by matching eigenfunction signatures built with histograms, and the retained set provides a smart initialization for the alignment problem with a considerable impact on the overall performance. Dense shape matching casted into graph matching reduces then, to point registration of embeddings under orthogonal transformations; the registration is solved using the framework of unsupervised clustering and the EM algorithm. Maximal subset matching of non identical shapes is handled by defining an appropriate outlier class. Experimental results on challenging examples show how the algorithm naturally treats changes of topology, shape variations and different sampling densities.

In this paper we propose a general framework to solve the articulated shape matching problem, formulated as finding point-to-point correspondences between two shapes represented by 2-D or 3-D point clouds. The original point- sets are embedded in a spectral representation and the actual matching is carried out in the embedded space. We analyze the advantages of this choice as well as the reasons for which the task remains a difficult one. In particular, we show that although embedded-space matching still has intrinsic combinatorial difficulties, it can be solved by searching for an optimal orthogonal transformation that aligns the two shape embeddings. Relying on the model based clustering formalism, we propose a probabilistic formulation which casts the matching into an EM algorithm. Outliers are properly handled by the algorithm and a simple strategy is adopted to initialize it. Experiments are performed with three embedding methods (Isomap, LLE, and Laplacian embedding) and with 3-D voxelsets representing a human-motion sequence.

In this paper we propose a method for matching articulated shapes represented as large sets of 3D points by aligning the corresponding embedded clouds generated by locally linear embedding. In particular we show that the problem is equivalent to aligning two sets of points under an orthogonal transformation acting onto the d-dimensional embeddings. The method may well be viewed as belonging to the model-based clustering framework and is implemented as an EM algorithm that alternates between the estimation of correspondences between data-points and the estimation of an optimal alignment transformation. Correspondences are initialized by embedding one set of data- points onto the other one through out-of-sample extension. Results for pairs of voxelsets representing moving persons are presented. Empirical evidence on the influence of the dimension of the embedding space is provided, suggesting that working with higher-dimensional spaces helps matching in challenging real-world scenarios, without collateral effects on the convergence.

Scene flow represents the 3-D motion of points in the scene, just as optical flow is related to their 2-D motion in the images. As opposed to classical methods which compute scene flow from optical flow, we propose to compute it by tracking 3-D points and surface elements (surfels) in a multi-camera setup (at least two cameras are needed). Two methods are proposed: in the first one, the translation of each 3-D point is found by matching the neighborhoods of its 2-D projections in each camera between two time steps; in the second one, the full pose of a surfel is recovered by matching the image of its projection with a texture template attached to the surfel, and visibility changes caused by occlusion or rotation of surfels are handled. Both methods detect lost or untrackable points and surfels. They were designed for real-time execution and can be used for fast extraction of scene flow from multi-camera sequences.

This work describes a navigation framework for robots in semi-structured outdoor environments which enables planning of semantic tasks by chaining of elementary visual-based movement primitives. Navigation is achieved by understanding the underlying world behind the image and using these results as a guideline to control the robot. As retrieving semantic information from vision is computationally demanding, short-term tasks are planned and executed while new vision information is processed. Thanks to learning techniques, the methods are adapted to different environment conditions. Fusion and filtering techniques provide reliability and stability to the system. The procedures have been fully integrated and tested with a real robot in an experimental environment. Results are discussed.