While representation learning and generative modeling seek to understand visual data, unifying both domains remains unexplored. Recent Unified Self-Supervised Learning (SSL) methods have started to bridge the gap between both paradigms. However, they either extract information from discriminative pretrained models or rely solely on semantic token reconstruction, which requires an external tokenizer during training — introducing a significant computational overhead. In this work, we introduce Sorcen, a novel Unified SSL framework, incorporating a synergic Contrastive-Reconstruction objective. Our novel Contrastive objective, leverages the generative capabilities of Sorcen and eliminates the need for additional image crops or augmentations during training. Sorcen "generates" contrastive positive samples, called Echoes, directly in the semantic token space using the reconstruction objective. This on-the-fly Echo generation, enables Sorcen to operate exclusively on precomputed tokens, eliminating the need for an online tokenizer during training. Sorcen significantly reduces the computational overhead by 60.8% compared to token reconstruction SoTA. Extensive experiments on ImageNet-1k demonstrate that Sorcen outperforms the previous Unified SSL SoTA by 0.4%, 1.48 FID, 1.76%, and 1.53% on linear probing, unconditional image generation, few-shot learning, and transfer learning, respectively. Additionally, Sorcen establishes as a new single-crop MIM SoTA in linear probing and achieves SoTA performance in unconditional image generation, highlighting significant improvements and breakthroughs in Unified SSL models1.

• Precision at Scale (PaS) automatically creates domain-specific datasets on-demand. • It leverages LLMs, VLMs, and generative models for data collection and curation. • PaS includes a task-agnostic framework for assessing dataset diversity. • Training on PaS datasets outperform model pretraining on large-scale general-domain datasets. • PaS datasets efficiently fine-tune SoTA VLMs in specialized domains. Recent self-supervised learning methods rely on massive general-domain datasets for robust model pretraining. However, these datasets may lack specificity required in specialized domains. Collecting large, supervised datasets to compensate for this limitation is also cumbersome. This raises a key question: Can automatically crafted domain-specific datasets serve as efficient and effective SSL pretrainers, performing comparable to—or even surpassing—much larger state-of-the-art general-domain datasets? To address this challenge, we propose Precision at Scale (PaS) , a novel modular pipeline for automatic creation of domain-specific datasets on-demand. PaS leverages Large Language Models (LLMs) and Vision-Language Models (VLMs) through three distinct phases: Concept Generation, where LLMs identify relevant domain concepts; Image Collection, utilizing VLMs and Generative models to gather appropriate images; Data Curation, ensuring quality and relevance by eliminating unrelated or redundant images. We conduct extensive experiments across three complex domains — food, insects, and birds — proving that PaS datasets compete and often surpass existing domain-specific datasets in diversity, scale, and effectiveness as pretrainers. Models pretrained on PaS datasets outperform those trained on large-scale general-domain datasets (ImageNet-1K) by up to 21 % and surpass same-scale domain-specific datasets by 6.7 % across classification tasks. Notably, despite being an order of magnitude smaller, PaS datasets outperform ImageNet-21K pretraining, with improvements of 3.3 % in fine-tuning and 9.5 % in few-shot learning, and showing superior performance on specialized dense tasks. Furthermore, by efficiently fine-tuning pretrained VLMs like CLIP and SigLIP using low-rank methods, we achieve performance gains (+4.2 % over CLIP) in specialized domains with minimal overhead, demonstrating the versatility of PaS datasets.

Food segmentation, including in videos, is vital for addressing real-world health, agriculture, and food biotechnology issues. Current limitations lead to inaccurate nutritional analysis, inefficient crop management, and suboptimal food processing, impacting food security and public health. Improving segmentation techniques can enhance dietary assessments, agricultural productivity, and the food production process. This study introduces the development of a robust framework for high-quality, near-real-time segmentation and tracking of food items in videos, using minimal hardware resources. We present FoodMem, a novel framework designed to segment food items from video sequences of 360-degree unbounded scenes. FoodMem can consistently generate masks of food portions in a video sequence, overcoming the limitations of existing semantic segmentation models, such as flickering and prohibitive inference speeds in video processing contexts. To address these issues, FoodMem leverages a two-phase solution: a transformer segmentation phase to create initial segmentation masks and a memory-based tracking phase to monitor food masks in complex scenes. Our framework outperforms current state-of-the-art food segmentation models, yielding superior performance across various conditions, such as camera angles, lighting, reflections, scene complexity, and food diversity. 2 2 More details in the supplementary material. This results in reduced segmentation noise, elimination of artifacts, and completion of missing segments. We also introduce a new annotated food dataset encompassing challenging scenarios absent in previous benchmarks. Extensive experiments conducted on MetaFood3D, Nutrition5k, and Vegetables & Fruits datasets demonstrate that FoodMem enhances the state-of-the-art by 2.5% mean average precision in food video segmentation and is 58 × faster on average. The source code is available at: 3 3 https://amughrabi.github.io/foodmem . . • Introduces FoodMem, the first near-real-time food video segmentation framework. • Leverages a segmentation transformer and memory model for mask refinement. • Outperforms FoodSAM, the state-of-the-art, across diverse conditions with 2.5% higher mAP. • Achieves 58x faster processing with minimal hardware resources. • Provides a novel annotated food dataset with challenging scenarios.

Current genotype-to-phenotype models, such as polygenic risk scores, only account for linear relationships between genotype and phenotype and ignore epistatic interactions, limiting the complexity of the diseases that can be properly characterized. Protein-protein interaction networks have the potential to improve the performance of the models. Moreover, interactions at the protein level can have profound implications in understanding the genetic etiology of diseases and, in turn, for drug development. In this article, we propose a novel approach for phenotype prediction based on graph neural networks (GNNs) that naturally incorporates existing protein interaction networks into the model. As a result, our approach can naturally discover relevant epistatic interactions. We assess the potential of this approach using simulations and comparing it to linear and other non-linear approaches. We also study the performance of the proposed GNN-based methods in predicting Alzheimer’s disease, one of the most complex neurodegenerative diseases, where our GNN approach outperform state of the art methods. In addition, we show that our proposal is able to discover critical interactions in the Alzheimer’s disease. Our findings highlight the potential of GNNs in predicting phenotypes and discovering the underlying mechanisms of complex diseases.

Leveraging inexpensive and human intervention-based annotating methodologies, such as crowdsourcing and web crawling, often leads to datasets with noisy labels. Noisy labels can have a detrimental impact on the performance and generalization of deep neural networks. Robust models that are able to handle and mitigate the effect of these noisy labels are thus essential. In this work, we explore the open challenges of neural network memorization and uncertainty in creating robust learning algorithms with noisy labels. To overcome them, we propose a novel framework called "Bayesian DivideMix++" with two critical components: (i) DivideMix++, to enhance the robustness against memorization and (ii) Monte-Carlo MixMatch, which focuses on improving the effectiveness towards label uncertainty. DivideMix++ improves the pipeline by integrating the warm-up and augmentation pipeline with self-supervised pre-training and dedicated different data augmentations for loss analysis and backpropagation. Monte-Carlo MixMatch leverages uncertainty measurements to mitigate the influence of uncertain samples by reducing their weight in the data augmentation MixMatch step. We validate our proposed pipeline using four datasets encompassing various synthetic and real-world noise settings. We demonstrate the effectiveness and merits of our proposed pipeline using extensive experiments. Bayesian DivideMix++ outperforms the state-of-the-art models by considerable differences in all experiments. Our findings underscore the potential of leveraging these modifications to enhance the performance and generalization of deep neural networks in practical scenarios.

The creation of large-scale datasets annotated by humans inevitably introduces noisy labels, leading to reduced generalization in deep-learning models. Sample selection-based learning with noisy labels is a recent approach that exhibits promising upbeat performance improvements. The selection of clean samples amongst the noisy samples is an important criterion in the learning process of these models. In this work, we delve deeper into the clean-noise split decision and highlight the aspect that effective demarcation of samples would lead to better performance. We identify the Global Noise Conundrum in the existing models, where the distribution of samples is treated globally. We propose a per-class-based local distribution of samples and demonstrate the effectiveness of this approach in having a better clean-noise split. We validate our proposal on several benchmarks - both real and synthetic, and show substantial improvements over different state-of-the-art algorithms. We further propose a new metric, classiness to extend our analysis and highlight the effectiveness of the proposed method. Source code and instructions to reproduce this paper are available at https://github.com/aldakata/CCLM/

Explainable Artificial Intelligence (XAI) provides tools to help understanding how AI models work and reach a particular decision or outcome. It helps to increase the interpretability of models and makes them more trustworthy and transparent. In this context, many XAI methods have been proposed to make black-box and complex models more digestible from a human perspective. However, one of the main issues that XAI methods have to face especially when dealing with a high number of features is the presence of multicollinearity, which casts shadows on the robustness of the XAI outcomes, such as the ranking of informative features. Most of the current XAI methods either do not consider the collinearity or assume the features are independent which, in general, is not necessarily true. Here, we propose a simple, yet useful, proxy that modifies the outcome of any XAI feature ranking method allowing to account for the dependency among the features, and to reveal their impact on the outcome. The proposed method was applied to SHAP, as an example of XAI method which assume that the features are independent. For this purpose, several models were exploited for a well-known classification task (males versus females) using nine cardiac phenotypes extracted from cardiac magnetic resonance imaging as features. Principal component analysis and biological plausibility were employed to validate the proposed method. Our results showed that the proposed proxy could lead to a more robust list of informative features compared to the original SHAP in presence of collinearity.

Nearest neighbour-based methods have proved to be one of the most successful self-supervised learning (SSL) approaches due to their high generalization capabilities. However, their computational efficiency decreases when more than one neighbour is used. In this paper, we propose a novel contrastive SSL approach, which we call All4One, that reduces the distance between neighbour representations using "centroids" created through a self-attention mechanism. We use a Centroid Contrasting objective along with single Neighbour Contrasting and Feature Contrasting objectives. Centroids help in learning contextual information from multiple neighbours whereas the neighbour contrast enables learning representations directly from the neighbours and the feature contrast allows learning representations unique to the features. This combination enables All4One to outperform popular instance discrimination approaches by more than 1% on linear classification evaluation for popular benchmark datasets and obtains state-of-the-art (SoTA) results. Finally, we show that All4One is robust towards embedding dimensionalities and augmentations, surpassing NNCLR and Barlow Twins by more than 5% on low dimensionality and weak augmentation settings. Source code is available in https://github.com/ImaGonEs/all4one.

Deep neural networks represent a compelling technique to tackle complex real-world problems, but are over-parameterized and often suffer from over- or under-confident estimates. Deep ensembles have shown better parameter estimations and often provide reliable uncertainty estimates that contribute to the robustness of the results. In this work, we propose a new metric to identify samples that are hard to classify. Our metric is defined as coincidence score for deep ensembles which measures the agreement of its individual models. The main hypothesis we rely on is that deep learning algorithms learn the low-loss samples better compared to large-loss samples. In order to compensate for this, we use controlled over-sampling on the identified ”hard” samples using proper data augmentation schemes to enable the models to learn those samples better. We validate the proposed metric using two public food datasets on different backbone architectures and show the improvements compared to the conventional deep neural network training using different performance metrics.

The automatic and accurate analysis of medical images (e.g., segmentation,detection, classification) are prerequisites for modern disease diagnosis and prognosis. Computer-aided diagnosis (CAD) systems empower accurate and effective detection of various diseases and timely treatment decisions. The past decade witnessed a spur in deep learning (DL)-based CADs showing outstanding performance across many health care applications. Medical imaging is hindered by multiple sources of uncertainty ranging fromnteasurement (aleatoric) errors, physiological variability, and limited medical knowledge (epistemic errors). However, uncertainty quantification (UQ) in most existing DL methods is insufficiently investigated, particularly in medical image analysis. Therefore, to address this gap, in this article, we propose a simple yet novel hierarchical attentive multilevel feature fusion model with an uncertainty-aware module for medical image classification coinedHercules. This approach is tested on several real medical image classification challenges. The proposedHerculesmodel consists of two main feature fusion blocks, where the former concentrates on attention-based fusion with uncertainty quantification module and the latter uses the raw features.Herculeswas evaluated across three medical imaging datasets, i.e., retinal OCT, lung CT, and chest X-ray.Herculesproduced the best classification accuracy in retinal OCT (94.21%), lung CT (99.59%), and chest X-ray (96.50%) datasets, respectively, against other state-of-the-art medical image classification methods.

Given the hyper-spectral imaging unique potentials in grasping the polymer characteristics of different materials, it is commonly used in sorting procedures. In a practical plastic sorting scenario, multiple plastic flakes may overlap which depending on their characteristics, the overlap can be reflected in their spectral signature. In this work, we use hyper-spectral imaging for the segmentation of three types of plastic flakes and their possible overlapping combinations. We propose an intuitive and simple multi-label encoding approach, bitfield encoding, to account for the overlapping regions. With our experiments, we show that the bitfield encoding improves over the baseline single-label approach and we further demonstrate its potential in predicting multiple labels for overlapping classes even when the model is only trained with non-overlapping classes.

Individuals diagnosed with Mild Cognitive Impairment (MCI) have shown an increased risk of developing Alzheimer’s Disease (AD). As such, early identification of dementia represents a key prognostic element, though hampered by complex disease patterns. Increasing efforts have focused on Machine Learning (ML) to build accurate classification models relying on a multitude of clinical/imaging variables. However, ML itself does not provide sensible explanations related to the model mechanism and feature contribution. Explainable Artificial Intelligence (XAI) represents the enabling technology in this framework, allowing to understand ML outcomes and derive human-understandable explanations. In this study, we aimed at exploring ML combined with MRI-based features and XAI to solve this classification problem and interpret the outcome. In particular, we propose a new method to assess the robustness of feature rankings provided by XAI methods, especially when multicollinearity exists. Our findings indicate that our method was able to disentangle the list of the informative features underlying dementia, with important implications for aiding personalized monitoring plans.

Deep learning methods show great promise in a range of settings including the biomedical field. Explainability of these models is important in these fields for building end-user trust and to facilitate their confident deployment. Although several Machine Learning Interpretability tools have been proposed so far, there is currently no recognized evaluation standard to transfer the explainability results into a quantitative score. Several measures have been proposed as proxies for quantitative assessment of explainability methods. However, the robustness of the list of significant features provided by the explainability methods has not been addressed. In this work, we propose a new proxy for assessing the robustness of the list of significant features provided by two explainability methods. Our validation is defined at functionality-grounded level based on the ranked correlation statistical index and demonstrates its successful application in the framework of brain aging estimation. We assessed our proxy to estimate brain age using neuroscience data. Our results indicate small variability and high robustness in the considered explainability methods using this new proxy.

Nutrition and social interactions are both key aspects of the daily lives of humans. In this work, we propose a system to evaluate the influence of social interaction in the nutritional habits of a person from a first-person perspective. In order to detect the routine of an individual, we construct a nutritional behaviour pattern discovery model, which outputs routines over a number of days. Our method evaluates similarity of routines with respect to visited food-related scenes over the collected days, making use of Dynamic Time Warping, as well as considering social engagement and its correlation with food-related activities. The nutritional and social descriptors of the collected days are evaluated and encoded using an LSTM Autoencoder. Later, the obtained latent space is clustered to find similar days unaffected by outliers using the Isolation Forest method. Moreover, we introduce a new score metric to evaluate the performance of the proposed algorithm. We validate our method on 104 days and more than 100 k egocentric images gathered by 7 users. Several different visualizations are evaluated for the understanding of the findings. Our results demonstrate good performance and applicability of our proposed model for social-related nutritional behaviour understanding. At the end, relevant applications of the model are discussed by analysing the discovered routine of particular individuals.

The determination of precise skin lesion boundaries in dermoscopic images using automated methods faces many challenges, most importantly, the presence of hair, inconspicuous lesion edges and low contrast in dermoscopic images, and variability in the color, texture and shapes of skin lesions. Existing deep learning-based skin lesion segmentation algorithms are expensive in terms of computational time and memory. Consequently, running such segmentation algorithms requires a powerful GPU and high bandwidth memory, which are not available in dermoscopy devices. Thus, this article aims to achieve precise skin lesion segmentation with minimum resources: a lightweight, efficient generative adversarial network (GAN) model called SLSNet, which combines 1-D kernel factorized networks, position and channel attention, and multiscale aggregation mechanisms with a GAN model. The 1-D kernel factorized network reduces the computational cost of 2D filtering. The position and channel attention modules enhance the discriminative ability between the lesion and non-lesion feature representations in spatial and channel dimensions, respectively. A multiscale block is also used to aggregate the coarse-to-fine features of input skin images and reduce the effect of the artifacts. SLSNet is evaluated on two publicly available datasets: ISBI 2017 and the ISIC 2018. Although SLSNet has only 2.35 million parameters, the experimental results demonstrate that it achieves segmentation results on a par with the state-of-the-art skin lesion segmentation methods with an accuracy of 97.61%, and Dice and Jaccard similarity coefficients of 90.63% and 81.98%, respectively. SLSNet can run at more than 110 frames per second (FPS) in a single GTX1080Ti GPU, which is faster than well-known deep learning-based image segmentation models, such as FCN. Therefore, SLSNet can be used for practical dermoscopic applications.

Recently, adversarial examples become one of the most dangerous risks in deep learning, which affects applications of real world such as robotics, cyber-security and computer vision. In image classification, adversarial attacks showed the ability to fool classifiers with small imperceptible perturbations added to the input. In this paper, we present an efficient defense mechanism, we call DVAE-SR that combine variational autoencoder and super-resolution to eliminate adversarial perturbation from image input before feeding it to the CNN classifier. The DVAE-SR can successfully defend against both white-box and black-box attacks without retraining CNN classifier and it recovers better accuracy than Defense-GAN and Defense-VAE.

Food recognition has recently attracted attention of many researchers. However, high food ambiguity, inter-class variability and intra-class similarity define a real challenge for the Deep learning and Computer Vision algorithms. In order to improve their performance, it is necessary to better understand what the model learns and, from this, to determine the type of data that should be additionally included for being the most beneficial to the training procedure. In this paper, we propose a new data augmentation strategy that estimates and uses the epistemic uncertainty to guide the model training. The method follows an active learning framework, where the new synthetic images are generated from the hard to classify real ones present in the training data based on the epistemic uncertainty. Hence, it allows the food recognition algorithm to focus on difficult images in order to learn their discriminatives features. On the other hand, avoiding data generation from images that do not contribute to the recognition makes it faster and more efficient. We show that the proposed method allows to improve food recognition and provides a better trade-off between micro- and macro-recall measures.

In this paper, we propose a new approach to understand actions in egocentric videos that exploit the semantics of object interactions at both frame and temporal levels. At the frame level, we use a region-based approach that takes as input a primary region roughly corresponding to the user hands and a set of secondary regions potentially corresponding to the interacting objects and calculates the action score through a CNN formulation. This information is then fed to a Hierarchical Long Short-Term Memory Network (HLSTM) that captures temporal dependencies between actions within and across shots. Ablation studies thoroughly validate the proposed approach, showing in particular that both levels of the HLSTM architecture contribute to performance improvement. Furthermore, quantitative comparisons show that the proposed approach outperforms the state-of-the-art in terms of action recognition on standard benchmarks, without relying on motion information.

With people living longer than ever, the number of cases with dementia such as Alzheimer's disease increases steadily. It affects more than 46 million people worldwide, and it is estimated that in 2050 more than 100 million will be affected. While there are no effective treatments for these terminal diseases, therapies such as reminiscence, that stimulate memories from the past are recommended. Currently, reminiscence therapy takes place in care homes and is guided by a therapist or a carer. In this work, we present an AI-based solution to automate the reminiscence therapy. This consists of a dialogue system that uses photos of the users as input to generate questions about their life. Overall, this paper presents how reminiscence therapy can be automated by using deep learning, and deployed to smartphones and laptops, making the therapy more accessible to every person affected by dementia.

Diagnostic medical imaging plays an imperative role in clinical assessment and treatment of medical abnormalities. The fusion of multimodal medical images merges complementary information present in the multi‐source images and provides a better interpretation with improved diagnostic accuracy. This paper presents a CT–MR neurological image fusion method using an optimised biologically inspired neural network in nonsubsampled shearlet (NSST) domain. NSST decomposed coefficients are utilised to activate the optimised neural model using particle swarm optimisation method and to generate the firing maps. Low and high‐frequency NSST subbands get fused using max‐rule based on firing maps. In the optimisation process, a fitness function is evaluated based on spatial frequency and edge index of the resultant fused image. To analyse the fusion performance, extensive experiments are conducted on the different CT–MR neurological image dataset. Objective performance is evaluated based on different metrics to highlight the clarity, contrast, correlation, visual quality, complementary information, salient information, and edge information present in the fused images. Experimental results show that the proposed method is able to provide better‐fused images and outperforms other existing methods in both visual and quantitative assessments.

Developing tools to understand and visualize lifestyle is of high interest when addressing the improvement of habits and well-being of people. Routine, defined as the usual things that a person does daily, helps describe the individuals’ lifestyle. With this paper, we are the first ones to address the development of novel tools for automatic discovery of routine days of an individual from his/her egocentric images. In the proposed model, sequences of images are firstly characterized by semantic labels detected by pre-trained CNNs. Then, these features are organized in temporal-semantic documents to later be embedded into a topic models space. Finally, Dynamic-Time-Warping and Spectral-Clustering methods are used for final day routine/non-routine discrimination. Moreover, we introduce a new EgoRoutine-dataset, a collection of 104 egocentric days with more than 100.000 images recorded by 7 users. Results show that routine can be discovered and behavioural patterns can be observed.

Recent studies have shown that the environment where people eat can affect their nutritional behavior [1]. In this paper, we provide automatic tools for personalized analysis of a person's health habits by the examination of daily recorded egocentric photo-streams. Specifically, we propose a new automatic approach for the classification of food-related environments, that is able to classify up to 15 such scenes. In this way, people can monitor the context around their food intake in order to get an objective insight into their daily eating routine. We propose a model that classifies food-related scenes organized in a semantic hierarchy. Additionally, we present and make available a new egocentric dataset composed of more than 33 000 images recorded by a wearable camera, over which our proposed model has been tested. Our approach obtains an accuracy and F-score of 56% and 65%, respectively, clearly outperforming the baseline methods.

This paper proposes an approach to automatically categorize the social interactions of a user wearing a photo-camera (2fpm), by relying solely on what the camera is seeing. The problem is challenging due to the overwhelming complexity of social life and the extreme intra-class variability of social interactions captured under unconstrained conditions. We adopt the formalization proposed in Bugental's social theory, that groups human relations into five social domains with related categories. Our method is a new deep learning architecture that exploits the hierarchical structure of the label space and relies on a set of social attributes estimated at frame level to provide a semantic representation of social interactions. Experimental results on the new EgoSocialRelation dataset demonstrate the effectiveness of our proposal.

The increase in awareness of people towards their nutritional habits has drawn considerable attention to the field of automatic food analysis. Focusing on self-service restaurants environment, automatic food analysis is not only useful for extracting nutritional information from foods selected by customers, it is also of high interest to speed up the service solving the bottleneck produced at the cashiers in times of high demand. In this paper, we address the problem of automatic food tray analysis in canteens and restaurants environment, which consists in predicting multiple foods placed on a tray image. We propose a new approach for food analysis based on convolutional neural networks, we name Semantic Food Detection, which integrates in the same framework food localization, recognition and segmentation. We demonstrate that our method improves the state of the art food detection by a considerable margin on the public dataset UNIMIB2016 achieving about 90% in terms of F-measure, and thus provides a significant technological advance towards the automatic billing in restaurant environments.

In our society and century, clothing is not anymore used only as a means for body protection. Our paper builds upon the evidence, studied within the social sciences, that clothing brings a clear communicative message in terms of social signals, influencing the impression and behaviour of others towards a person. In fact, clothing correlates with personality traits, both in terms of self-assessment and assessments that unacquainted people give to an individual. The consequences of these facts are important: the influence of clothing on the decision making of individuals has been investigated in the literature, showing that it represents a discriminative factor to differentiate among diverse groups of people. Unfortunately, this has been observed after cumbersome and expensive manual annotations, on very restricted populations, limiting the scope of the resulting claims. With this position paper, we want to sketch the main steps of the very first systematic analysis, driven by social signal processing techniques, of the relationship between clothing and social signals, both sent and perceived. Thanks to human parsing technologies, which exhibit high robustness owing to deep learning architectures, we are now capable to isolate visual patterns characterising a large types of garments. These algorithms will be used to capture statistical relations on a large corpus of evidence to confirm the sociological findings and to go beyond the state of the art.

Given an unconstrained stream of images captured by a wearable photo-camera (2fpm), we propose an unsupervised bottom-up approach for automatic clustering appearing faces into the individual identities present in these data. The problem is challenging since images are acquired under real world conditions; hence the visible appearance of the people in the images undergoes intensive variations. Our proposed pipeline consists of first arranging the photo-stream into events, later, localizing the appearance of multiple people in them, and finally, grouping various appearances of the same person across different events. Experimental results performed on a dataset acquired by wearing a photo-camera during one month, demonstrate the effectiveness of the proposed approach for the considered purpose.

The organisation of personal data is receiving increasing research attention due to the challenges we face in gathering, enriching, searching, and visualising such data. Given the increasing ease with which personal data being gathered by individuals, the concept of a lifelog digital library of rich multimedia and sensory content for every individual is fast becoming a reality. The LTA 2017 workshop aims to bring together academics and practitioners to discuss approaches to lifelog data analytics and applications; and to debate the opportunities and challenges for researchers in this new and challenging area.

While wearable cameras are becoming increasingly popular, locating relevant information in large unstructured collections of egocentric images is still a tedious and time consuming process. This paper addresses the problem of organizing egocentric photo streams acquired by a wearable camera into semantically meaningful segments, hence making an important step towards the goal of automatically annotating these photos for browsing and retrieval. In the proposed method, first, contextual and semantic information is extracted for each image by employing a Convolutional Neural Networks approach. Later, a vocabulary of concepts is defined in a semantic space by relying on linguistic information. Finally, by exploiting the temporal coherence of concepts in photo streams, images which share contextual and semantic attributes are grouped together. The resulting temporal segmentation is particularly suited for further analysis, ranging from event recognition to semantic indexing and summarization. Experimental results over egocentric set of nearly 31,000 images, show the prominence of the proposed approach over state-of-the-art methods.

Visual lifelogging consists of acquiring images that capture the daily experiences of the user by wearing a camera over a long period of time. The pictures taken offer considerable potential for knowledge mining concerning how people live their lives; hence, they open up new opportunities for many potential applications in fields including healthcare, security, leisure, and the quantified self. However, automatically building a story from a huge collection of unstructured egocentric data presents major challenges. This paper provides a thorough review of advances made so far in egocentric data analysis and, in view of the current state of the art, indicates new lines of research to move us toward storytelling from visual lifelogging.

Characterization of carotid plaque composition, more specifically the amount of lipid core, fibrous tissue, and calcified tissue, is an important task for the identification of plaques that are prone to rupture, and thus for early risk estimation of cardiovascular and cerebrovascular events. Due to its low costs and wide availability, carotid ultrasound has the potential to become the modality of choice for plaque characterization in clinical practice. However, its significant image noise, coupled with the small size of the plaques and their complex appearance, makes it difficult for automated techniques to discriminate between the different plaque constituents. In this paper, we propose to address this challenging problem by exploiting the unique capabilities of the emerging deep learning framework. More specifically, and unlike existing works which require a priori definition of specific imaging features or thresholding values, we propose to build a convolutional neural network (CNN) that will automatically extract from the images the information that is optimal for the identification of the different plaque constituents. We used approximately 90 000 patches extracted from a database of images and corresponding expert plaque characterizations to train and to validate the proposed CNN. The results of cross-validation experiments show a correlation of about 0.90 with the clinical assessment for the estimation of lipid core, fibrous cap, and calcified tissue areas, indicating the potential of deep learning for the challenging task of automatic characterization of plaque composition in carotid ultrasound.

Although traditionally used in the machine translation field, the encoder-decoder framework has been recently applied for the generation of video and image descriptions. The combination of Convolutional and Recurrent Neural Networks in these models has proven to outperform the previous state of the art, obtaining more accurate video descriptions. In this work we propose pushing further this model by introducing two contributions into the encoding stage. First, producing richer image representations by combining object and location information from Convolutional Neural Networks and second, introducing Bidirectional Recurrent Neural Networks for capturing both forward and backward temporal relationships in the input frames.

Given a user wearing a low frame rate wearable camera during a day, this work aims to automatically detect the moments when the user gets engaged into a social interaction solely by reviewing the automatically captured photos by the worn camera. The proposed method, inspired by the sociological concept of F-formation, exploits distance and orientation of the appearing individuals -with respect to the user- in the scene from a bird-view perspective. As a result, the interaction pattern over the sequence can be understood as a two-dimensional time series that corresponds to the temporal evolution of the distance and orientation features over time. A Long-Short Term Memory-based Recurrent Neural Network is then trained to classify each time series. Experimental evaluation over a dataset of 30.000 images has shown promising results on the proposed method for social interaction detection in egocentric photo-streams.

The development of automatic nutrition diaries, which would allow to keep track objectively of everything we eat, could enable a whole new world of possibilities for people concerned about their nutrition patterns. With this purpose, in this paper we propose the first method for simultaneous food localization and recognition. Our method is based on two main steps, which consist in, first, produce a food activation map on the input image (i.e. heat map of probabilities) for generating bounding boxes proposals and, second, recognize each of the food types or food-related objects present in each bounding box. We demonstrate that our proposal, compared to the most similar problem nowadays - object localization, is able to obtain high precision and reasonable recall levels with only a few bounding boxes. Furthermore, we show that it is applicable to both conventional and egocentric images.

The organisation of personal data is receiving increasing research attention due to the challenges we face in gathering, enriching, searching, and visualising such data. Given the increasing ease with which personal data being gathered by individuals, the concept of a lifelog digital library of rich multimedia and sensory content for every individual is fast becoming a reality. The LTA~2016 workshop aims to bring together academics and practitioners to discuss approaches to lifelog data analytics and applications; and to debate the opportunities and challenges for researchers in this new and challenging area.

Detecting social interaction in videos relying solely on visual cues is a valuable task that is receiving increasing attention in recent years. In this work, we address this problem in the challenging domain of egocentric photo-streams captured by a low temporal resolution wearable camera (2fpm). The major difficulties to be handled in this context are the sparsity of observations as well as unpredictability of camera motion and attention orientation due to the fact that the camera is worn as part of clothing. Our method consists of four steps: multi-faces localization and tracking, 3D localization, pose estimation and analysis of f-formations. By estimating pair-to-pair interaction probabilities over the sequence, our method states the presence or absence of interaction with the camera wearer and specifies which people are more involved in the interaction. We tested our method over a dataset of 18.000 images and we show its reliability on our considered purpose.

We propose a meta-parameter free, off-the-shelf, simple and fast unsupervised feature learning algorithm, which exploits a new way of optimizing for sparsity. Experiments on CIFAR-10, STL-10 and UCMerced show that the method achieves the state-of-the-art performance, providing discriminative features that generalize well.

In this paper, we tackle the problem of unsupervised segmentation of intestinal events using various motility descriptors extracted from the endoluminal video. The segments of constant intestinal activity are detected with a robust statistical test that is based on Hoeffding's inequality. The qualitative analysis of the results shows that the segments adjust well to the motility information and thus, this segmentation can constitute basic units for higher-level motility description.

Intestinal contractions are one of the most important events to diagnose motility pathologies of the small intestine. When visualized by wireless capsule endoscopy (WCE), the sequence of frames that represents a contraction is characterized by a clear wrinkle structure in the central frames that corresponds to the folding of the intestinal wall. In this paper, we present a new method to robustly detect wrinkle frames in full WCE videos by using a new mid-level image descriptor that is based on a centrality measure proposed for graphs. We present an extended validation, carried out in a very large database, that shows that the proposed method achieves state-of-the-art performance for this task.

We present a generic framework for object segmentation using depth maps based on Random Forest and Graph-cuts theory, and apply it to the segmentation of human limbs in depth maps. First, from a set of random depth features, Random Forest is used to infer a set of label probabilities for each data sample. This vector of probabilities is used as unary term in α-β swap Graph-cuts algorithm. Moreover, depth of spatio-temporal neighboring data points are used as boundary potentials. Results on a new multi-label human depth data set show high performance in terms of segmentation overlapping of the novel methodology compared to classical approaches.

Segmentation of coronary arteries in X-Ray angiography is a fundamental tool to evaluate arterial diseases and choose proper coronary treatment. The accurate segmentation of coronary arteries has become an important topic for the registration of different modalities which allows physicians rapid access to different medical imaging information from Computed Tomography (CT) scans or Magnetic Resonance Imaging (MRI). In this paper, we propose an accurate fully automatic algorithm based on Graph-cuts for vessel centerline extraction, caliber estimation, and catheter detection. Vesselness, geodesic paths, and a new multi-scale edgeness map are combined to customize the Graph-cuts approach to the segmentation of tubular structures, by means of a global optimization of the Graph-cuts energy function. Moreover, a novel supervised learning methodology that integrates local and contextual information is proposed for automatic catheter detection. We evaluate the method performance on three datasets coming from different imaging systems. The method performs as good as the expert observer w.r.t. centerline detection and caliber estimation. Moreover, the method discriminates between arteries and catheter with an accuracy of 96.5%, sensitivity of 72%, and precision of 97.4%.

Wireless capsule endoscopy (WCE) is a device that allows the direct visualization of gastrointestinal tract with minimal discomfort for the patient, but at the price of a large amount of time for screening. In order to reduce this time, several works have proposed to automatically remove all the frames showing intestinal content. These methods label frames as {intestinal content- clear} without discriminating between types of content (with different physiological meaning) or the portion of image covered. In addition, since the presence of intestinal content has been identified as an indicator of intestinal motility, its accurate quantification can show a potential clinical relevance. In this paper, we present a method for the robust detection and segmentation of intestinal content in WCE images, together with its further discrimination between turbid liquid and bubbles. Our proposal is based on a twofold system. First, frames presenting intestinal content are detected by a support vector machine classifier using color and textural information. Second, intestinal content frames are segmented into {turbid, bubbles, and clear} regions. We show a detailed validation using a large dataset. Our system outperforms previous methods and, for the first time, discriminates between turbid from bubbles media.

In this paper, we propose a circular blurred shape model descriptor to deal with the problem of symbol detection and classification as a particular case of object recognition. The feature extraction is performed by capturing the spatial arrangement of significant object characteristics in a correlogram structure. The shape information from objects is shared among correlogram regions, where a prior blurring degree defines the level of distortion allowed in the symbol, making the descriptor tolerant to irregular deformations. Moreover, the descriptor is rotation invariant by definition. We validate the effectiveness of the proposed descriptor in both the multiclass symbol recognition and symbol detection domains. In order to perform the symbol detection, the descriptors are learned using a cascade of classifiers. In the case of multiclass categorization, the new feature space is learned using a set of binary classifiers which are embedded in an error-correcting output code design. The results over four symbol data sets show the significant improvements of the proposed descriptor compared to the state-of-the-art descriptors. In particular, the results are even more significant in those cases where the symbols suffer from elastic deformations.

We present a meta-learning framework for the design of potential functions for Conditional Random Fields. The design of both node potential and edge potential is formulated as a classification problem where margin classifiers are used. The set of state transitions for the edge potential is treated as a set of different classes, thus defining a multi-class learning problem. The Error-Correcting Output Codes (ECOC) technique is used to deal with the multi-class problem. Furthermore, the point defined by the combination of margin classifiers in the ECOC space is interpreted in a probabilistic manner, and the obtained distance values are then converted into potential values. The proposed model exhibits very promising results when applied to two real detection problems.

This article proposes a general extension of the Error Correcting Output Codes (ECOC) framework to the online learning scenario. As a result, the final classifier handles the addition of new classes independently of the base classifier used. Validation on UCI database and two real machine vision applications show that the online problem-dependent ECOC proposal provides a feasible and robust way for handling new classes using any base classifier.

A common way to model multiclass classification problems is to design a set of binary classifiers and to combine them. Error-Correcting Output Codes (ECOC) represent a successful framework to deal with these type of problems. Recent works in the ECOC framework showed significant performance improvements by means of new problem-dependent designs based on the ternary ECOC framework. The ternary framework contains a larger set of binary problems because of the use of a "do not care" symbol that allows us to ignore some classes by a given classifier. However, there are no proper studies that analyze the effect of the new symbol at the decoding step. In this paper, we present a taxonomy that embeds all binary and ternary ECOC decoding strategies into four groups. We show that the zero symbol introduces two kinds of biases that require redefinition of the decoding design. A new type of decoding measure is proposed, and two novel decoding strategies are defined. We evaluate the state-of-the-art coding and decoding strategies over a set of UCI Machine Learning Repository data sets and into a real traffic sign categorization problem. The experimental results show that, following the new decoding strategies, the performance of the ECOC design is significantly improved.

Bioabsorbable drug-eluting coronary stents present a very promising improvement to the common metallic ones solving some of the most important problems of stent implantation: the late restenosis. These stents made of poly-L-lactic acid cause a very subtle acoustic shadow (compared to the metallic ones) making difficult the automatic detection and measurements in images. In this paper, we propose a novel approach based on a cascade of GentleBoost classifiers to detect the stent struts using structural features to code the information of the different subregions of the struts. A stochastic gradient descent method is applied to optimize the overall performance of the detector. Validation results of struts detection are very encouraging with an average F-measure of 81%.

Intestinal motility assessment with video capsule endoscopy arises as a novel and challenging clinical fieldwork. This technique is based on the analysis of the patterns of intestinal contractions shown in a video provided by an ingestible capsule with a wireless micro-camera. The manual labeling of all the motility events requires large amount of time for offline screening in search of findings with low prevalence, which turns this procedure currently unpractical. In this paper, we propose a machine learning system to automatically detect the phasic intestinal contractions in video capsule endoscopy, driving a useful but not feasible clinical routine into a feasible clinical procedure. Our proposal is based on a sequential design which involves the analysis of textural, color, and blob features together with SVM classifiers. Our approach tackles the reduction of the imbalance rate of data and allows the inclusion of domain knowledge as new stages in the cascade. We present a detailed analysis, both in a quantitative and a qualitative way, by providing several measures of performance and the assessment study of interobserver variability. Our system performs at 70% of sensitivity for individual detection, whilst obtaining equivalent patterns to those of the experts for density of contractions.

Symbol spotting problem requires feature extraction strategies able to generalize from training samples and to localize the target object while discarding most part of the image. In the case of document analysis, symbol spotting techniques have to deal with a high variability of symbols' appearance. In this paper, we propose the Circular Blurred Shape Model descriptor. Feature extraction is performed capturing the spatial arrangement of significant object characteristics in a correlogram structure. Shape information from objects is shared among correlogram regions, being tolerant to the irregular deformations. Descriptors are learnt using a cascade of classifiers and Abadoost as the base classifier. Finally, symbol spotting is performed by means of a windowing strategy using the learnt cascade over plan and old musical score documents. Spotting and multi-class categorization results show better performance comparing with the state-of-the-art descriptors.

Ten years ago the concept of bilateral filtering (BF) became popular in the image processing community. The core of the idea is to blend the effect of a spatial filter, as e.g. the Gaussian filter, with the effect of a filter that acts on image values. The two filters acts on orthogonal domains of a picture: the 2D lattice of the image support and the intensity (or color) domain. The BF approach is an intuitive way to blend these two filters giving rise to algorithms that perform difficult tasks requiring a relatively simple design. In this paper we extend the concept of BF, proposing the bilateral enhancers (BE). We show how to design proper functions to obtain an edge-preserving smoothing and a selective sharpening. Moreover, we show that the proposed algorithm can perform edge-preserving smoothing and selective sharpening simultaneously in a single filtering.

Rich online map interaction represents a useful tool to get multimedia information related to physical places. With this type of systems, users can automatically compute the optimal route for a trip or to look for entertainment places or hotels near their actual position. Standard maps are defined as a fusion of layers, where each one contains specific data such height, streets, or a particular business location. In this paper we propose the construction of a visual content layer which describes the visual appearance of geographic locations in a city. We captured, by means of a mobile mapping system, a huge set of georeferenced images (> 500 K) which cover the whole city of Barcelona. For each image, hundreds of region descriptions are computed off-line and described as a hash code. This allows an efficient and scalable way of accessing maps by visual content.

Intravascular ultrasound (IVUS) technology permits visualization of high-resolution images of internal vascular structures. IVUS is a unique image-guiding tool to display longitudinal view of the vessels, and estimate the length and size of vascular structures with the goal of accurate diagnosis. Unfortunately, due to pulsatile contraction and expansion of the heart, the captured images are affected by different motion artifacts that make visual inspection difficult. In this paper, we propose an efficient algorithm that aligns vascular structures and strongly reduces the saw-shaped oscillation, simplifying the inspection of longitudinal cuts; it reduces the motion artifacts caused by the displacement of the catheter in the short-axis plane and the catheter rotation due to vessel tortuosity. The algorithm prototype aligns 3.16 frames/s and clearly outperforms state-of-the-art methods with similar computational cost. The speed of the algorithm is crucial since it allows to inspect the corrected sequence during patient intervention. Moreover, we improved an indirect methodology for IVUS rigid registration algorithm evaluation.

The high variability of sign appearance in uncontrolled environments has made the detection and classification of road signs a challenging problem in computer vision. In this paper, we introduce a novel approach for the detection and classification of traffic signs. Detection is based on a boosted detectors cascade, trained with a novel evolutionary version of Adaboost, which allows the use of large feature spaces. Classification is defined as a multiclass categorization problem. A battery of classifiers is trained to split classes in an Error-Correcting Output Code (ECOC) framework. We propose an ECOC design through a forest of optimal tree structures that are embedded in the ECOC matrix. The novel system offers high performance and better accuracy than the state-of-the-art strategies and is potentially better in terms of noise, affine deformation, partial occlusions, and reduced illumination.

Tissue biomechanical properties (like strain and stress) are playing an increasing role in diagnosis and long-term treatment of intravascular coronary diseases. Their assessment strongly relies on estimation of vessel wall deformation. Since intravascular ultrasound (IVUS) sequences allow visualizing vessel morphology and reflect its dynamics, this technique represents a useful tool for evaluation of tissue mechanical properties. Image misalignment introduced by vessel-catheter motion is a major artifact for a proper tracking of tissue deformation. In this work, we focus on compensating and assessing IVUS rigid in-plane motion due to heart beating. Motion parameters are computed by considering both the vessel geometry and its appearance in the image. Continuum mechanics laws serve to introduce a novel score measuring motion reduction in in vivo sequences. Synthetic experiments validate the proposed score as measure of motion parameters accuracy; whereas results in in vivo pullbacks show the reliability of the presented methodologies in clinical cases.

Error correcting output codes (ECOC) represent a successful framework to deal with multi-class categorization problems based on combining binary classifiers. In this paper, we present a new formulation of the ternary ECOC distance and the error-correcting capabilities in the ternary ECOC framework. Based on the new measure, we stress on how to design coding matrices preventing codification ambiguity and propose a new sparse random coding matrix with ternary distance maximization. The results on the UCI Repository and in a real speed traffic categorization problem show that when the coding design satisfies the new ternary measures, significant performance improvement is obtained independently of the decoding strategy applied.

The Chagas¿ disease is endemic in all Latin America, affecting millions of people in the continent. In order to diagnose and treat the Chagas¿ disease, it is important to detect and measure the coronary damage of the patient. In this paper, we analyze and categorize patients into different groups based on the coronary damage produced by the disease. Based on the features of the heart cycle extracted using high resolution ECG, a multi-class scheme of error-correcting output codes (ECOC) is formulated and successfully applied. The results show that the proposed scheme obtains significant performance improvements compared to previous works and state-of-the-art ECOC designs.

A common way to model multi-class classification problems is by means of Error-Correcting Output Codes (ECOC). Given a multi-class problem, the ECOC technique designs a code word for each class, where each position of the code identifies the membership of the class for a given binary problem. A classification decision is obtained by assigning the label of the class with the closest code. One of the main requirements of the ECOC design is that the base classifier is capable of splitting each sub-group of classes from each binary problem. However, we can not guarantee that a linear classifier model convex regions. Furthermore, non-linear classifiers also fail to manage some type of surfaces. In this paper, we present a novel strategy to model multi-class classification problems using sub-class information in the ECOC framework. Complex problems are solved by splitting the original set of classes into sub-classes, and embedding the binary problems in a problem-dependent ECOC design. Experimental results show that the proposed splitting procedure yields a better performance when the class overlap or the distribution of the training objects conceil the decision boundaries for the base classifier. The results are even more significant when one has a sufficiently large training size.

In this paper, we present a general guideline to find a better distance measure for similarity estimation based on statistical analysis of distribution models and distance functions. A new set of distance measures are derived from the harmonic distance, the geometric distance, and their generalized variants according to the Maximum Likelihood theory. These measures can provide a more accurate feature model than the classical Euclidean and Manhattan distances. We also find that the feature elements are often from heterogeneous sources that may have different influence on similarity estimation. Therefore, the assumption of single isotropic distribution model is often inappropriate. To alleviate this problem, we use a boosted distance measure framework that finds multiple distance measures which fit the distribution of selected feature elements best for accurate similarity estimation. The new distance measures for similarity estimation are tested on two applications: stereo matching and motion tracking in video sequences. The performance of boosted distance measure is further evaluated on several benchmark data sets from the UCI repository and two image retrieval applications. In all the experiments, robust results are obtained based on the proposed methods.

Despite recovering a normal coronary flow after acute myocardial infarction, percutaneous coronary intervention does not guarantee a proper perfusion (irrigation) of the infarcted area. This damage in microcirculation integrity may detrimentally affect the patient survival. Visual assessment of the myocardium opacification in contrast angiography serves to define a subjective score of the microcirculation integrity myocardial blush analysis (MBA). Although MBA correlates with patient prognosis its visual assessment is a very difficult task that requires of a highly expertise training in order to achieve a good intraobserver and interobserver agreement. In this paper, we provide objective descriptors of the myocardium staining pattern by analyzing the spectrum of the image local statistics. The descriptors proposed discriminate among the different phenomena observed in the angiographic sequence and allow defining an objective score of the myocardial perfusion.

This paper presents an efficient object-class recognition approach based on a new type of image descriptor: the Class-Specific Binary Correlogram (CSBC). In our representation, the image is described by a collection of CSBCs, where each one encodes the spatial distribution of class-specific features around a particular reference point. This representation is obtained by first performing an automatic selection of class-specific features from a vocabulary, and then extracting collections of binary correlograms that encode, at the same time, detected object parts and their spatial distribution around multiple points of the image. Our descriptors live in high-dimensional spaces (in the order of 10K dimensions), but they are very sparse. We show that efficient learning and matching procedures can be obtained for such a representation if we use, first, fast feature selection techniques specific for binary features, and then Boosting integrated with an appropriate Inverted File data organization. The proposed strategy works with weak supervision, outperforms state-of-the-art bag-of-feature methods, and it is more accurate and computationally more efficient than well-known geometrical-based methods, including our previous work on Generalized Correlograms (GCs) [1]. 1

The good of interest point detectors is to find, in an unsupervised way, keypoints easy to extract and at the same time robust to image transformations. We present a novel set of saliency feathers based on image singularities that takes into account the region content in terms of intensity and local structure. The region complexity is estimated by means of the entropy of the grey-level information; shape information is obtained by measuring the entropy of significant orientations. The regions are located in their representative scale and categorized by their complexity level. Thus, the regions are highly discriminable and less sensitive to confusion and false alalarm than the traditional approaches. We compare the novel complex salient regions with the state-of-the-art keypoint detectors. The presented interest points show robustness to a wide set of image transformations and high repeatability, as well as allows matching from different camera points of view. Beside. We show the temporal robustness of the novel salient regions in real video sequences, being potentially useful for matching, image retrieval, and object categorization problems.

We present a novel approach for retrieval of object categories based on a novel type of image representation: the Generalized Correlogram (GC). In our image representation, the object is described as a constellation of GCs where each one encodes information about some local part and the spatial relations from this part to others (i.e., the part's context). We show how such a representation can be used with fast procedures that learn the object category with weak supervision and efficiently match the model of the object against large collections of images. In the learning stage, we show that by integrating our representation with Boosting the system is able to obtain a compact model that is represented by very few features, where each feature conveys key properties about the object's parts and their spatial arrangement. In the matching step, we propose direct procedures that exploit our representation for efficiently considering spatial coherence between the matching of local parts. Combined with an appropriate data organization such as Inverted Files, we show that thousands of images can be evaluated efficiently. The framework has been applied to different standard databases and we show that our results are favorably compared against state-of-the-art methods in both computational cost and accuracy.

In this paper, a real-time application for visual inspection and classification of cork stoppers is presented. The process of cork inspection and quality grading is based on analyzing a large set of characteristics corresponding to visual features that are related to cork porosity. We have applied a set of nonparametric and parametric classification methods for comparing and evaluating their performance in this real problem. The best results have been achieved using Bayesian classification through probabilistic modeling in a high-dimensional space. In this context, it is well known that high dimensionality represents a serious problem for density estimation. We propose a class-conditional independent component analysis representation of the data that allows an accurate estimation of the data probability density function by factorizing it. The method has achieved a success of 98% of correct classification

We present a heuristic method for learning error correcting output codes matrices based on a hierarchical partition of the class space that maximizes a discriminative criterion. To achieve this goal, the optimal codeword separation is sacrificed in favor of a maximum class discrimination in the partitions. The creation of the hierarchical partition set is performed using a binary tree. As a result, a compact matrix with high discrimination power is obtained. Our method is validated using the UCI database and applied to a real problem, the classification of traffic sign images.

Vessel plaque assessment by analysis of intravascular ultrasound sequences is a useful tool for cardiac disease diagnosis and intervention. Manual detection of luminal (inner) and media-adventitia (external) vessel borders is the main activity of physicians in the process of lumen narrowing (plaque) quantification. Difficult definition of vessel border descriptors, as well as, shades, artifacts, and blurred signal response due to ultrasound physical properties trouble automated adventitia segmentation. In order to efficiently approach such a complex problem, we propose blending advanced anisotropic filtering operators and statistical classification techniques into a vessel border modelling strategy. Our systematic statistical analysis shows that the reported adventitia detection achieves an accuracy in the range of interobserver variability regardless of plaque nature, vessel geometry, and incomplete vessel borders.

We present a novel approach for fast object class recognition incorporating contextual information into boosting. The object is represented as a constellation of generalized correlograms that integrate both information of local parts and their spatial relations. Incorporating the spatial relations into our constellation of descriptors, we show that an exhaustive search for the best matching can be avoided. Combining the contextual descriptors with boosting, the system simultaneously learns the information that characterize each part of the object along with their characteristic mutual spatial relations. The proposed framework includes a matching step between homologous parts in the training set, and learning the spatial pattern after matching. In the matching part two approaches are provided: a supervised algorithm and an unsupervised one. Our results are favorably compared against state-of-the-art results.

We present a content-based image retrieval system of medical images of bodies with high elasticity, that is, high inter- and intrasubject variability in shape. The system is based on a rich feature space that is able to describe all the relevant aspects of an image, including local, global, and contextual information. For including relative spatial relations between the structures (i.e., contextual information) we do not need to obtain very accurate segmentations of the image, in contrast to the majority of methods employed for this kind of description. We also obtain invariance to spatial deformations of the same type of object along different instances. This is achieved by applying efficient registrations of the images before their comparison. The incorporation of all these components represents an innovative and powerful way of comparing and retrieving medical images. Validated results are reported on a database of 168 intravascular ultrasound images, showing the appropriateness of our approach for images of such high complexity. © 2005 Wiley Periodicals, Inc. Int J Int Syst 20: 541–559, 2005.

Among all filtering techniques, those based exclusively on image level sets (geometric flows) have proven to be the less sensitive to the nature of noise and the most contrast preserving. A common feature to existent curvature flows is that they penalize high curvature, regardless of the curve regularity. This constitutes a major drawback since curvature extreme values are standard descriptors of the contour geometry. We argue that an operator designed with shape recovery purposes should include a term penalizing irregularity in the curvature rather than its magnitude. To this purpose, we present a novel geometric flow that includes a function that measures the degree of local irregularity present in the curve. A main advantage is that it achieves non-trivial steady states representing a smooth model of level curves in a noisy image. Performance of our approach is compared to classical filtering techniques in terms of quality in the restored image/shape and asymptotic behavior. We empirically prove that our approach is the technique that achieves the best compromise between image quality and evolution stabilization.

In this paper we introduce a novel application of the diffusion tensor for anisotropic image processing. The anisotropic contour completion (ACC) we suggest consists in extending the characteristic function of the open curve by means of a degenerated diffusion tensor that prevents any diffusion in the normal direction. We show that ACC is equivalent to a dilation with a continuous elliptic structural element that takes into account the local orientation of the contours to be closed. Experiments on contours extracted from real images show that ACC produces shapes able to adapt to any curve in an active contour framework.

This paper is concerned with the three-dimensional (3-D) reconstruction of coronary vessel centerlines and with how distortion of X-ray angiographic images affects it. Angiographies suffer from pincushion and other geometrical distortions, caused by the peripheral concavity of the image intensifier (II) and the nonlinearity of electronic acquisition devices. In routine clinical practice, where a field-of-view (FOV) of 17-23 cm is commonly used for the acquisition of coronary vessels, this distortion introduces a positional error of up to 7 pixels for an image matrix size of 512 x 512 and an FOV of 17 cm. This error increases with the size of the FOV. Geometrical distortions have a significant effect on the validity of the 3-D reconstruction of vessels from these images. We show how this effect can be reduced by integrating a predictive model of (un)distortion into the biplane snakes formulation for 3-D reconstruction. First, we prove that the distortion can be accurately modeled using a polynomial for each view. Also, we show that the estimated polynomial is independent of focal length, but not of changes in anatomical angles, as the II is influenced by the earth's magnetic field. Thus, we decompose the polynomial into two components: the steady and the orientation-dependent component. We determine the optimal polynomial degree for each component, which is empirically determined to be five for the steady component and three for the orientation-dependent component. This fact simplifies the prediction of the orientation-dependent polynomial, since the number of polynomial coefficients to be predicted is lower. The integration of this model into the biplane snakes formulation enables us to avoid image unwarping, which deteriorates image quality and therefore complicates vessel centerline feature extraction. Moreover, we improve the biplane snake behavior when dealing with wavy vessels, by means of using generalized gradient vector flow. Our experiments show that the proposed methods in this paper decrease up to 88% the reconstruction error obtained when geometrical distortion effects are ignored. Tests on imaged phantoms and real cardiac images are presented as well.

This paper proposes a statistical framework for segmenting textured areas over real images by discriminant snakes. Our active contour model has the ability to learn different texture prototypes and generate a global statistical model from a multi-valued function. This function is generated by means of filter responses over the texture regions. Linear discriminant analysis is performed to obtain a statistical classifier embodied into the snake scheme. Given an input image composed of different texture types, a likelihood map is built and the discriminant snake deforms on it to delineate regions with similar texture descriptions according to the learned texture patterns. Our method is tested on two different image applications: aerial images and medical (ultrasound) images, and the results are very encouraging.

Magnetic resonance imaging (MRI) is unique in its ability to noninvasively and selectively alter tissue magnetization, and create tag planes intersecting image slices. The resulting grid of signal voids allows for tracking deformations of tissues in otherwise homogeneous-signal myocardial regions. In this paper, we propose a specific spatial modulation of magnetization (SPAMM) imaging protocol together with efficient techniques for measurement of three-dimensional (3-D) motion of material points of the human heart (referred to as myocardial beads) from images collected with the SPAMM method. The techniques make use of tagged images in orthogonal views by explicitly reconstructing 3-D B-spline surface representation of tag planes (tag planes in two orthogonal orientations intersecting the short-axis (SA) image slices and tag planes in an orientation orthogonal to the short-axis tag planes intersecting long-axis (LA) image slices). The developed methods allow for viewing deformations of 3-D tag surfaces, spatial correspondence of long-axis and short-axis image slice and tag positions, as well as nonrigid movement of myocardial beads as a function of time.

In this paper we introduce a statistic snake that learns and tracks image features by means of statistic learning techniques. Using probabilistic principal component analysis a feature description is obtained from a training set of object profiles. In our approach a sound statistical model is introduced to define a likelihood estimate of the grey-level local image profiles together with their local orientation. This likelihood estimate allows to define a probabilistic potential field of the snake where the elastic curve deforms to maximise the overall probability of detecting learned image features. To improve the convergence of snake deformation, we enhance the likelihood map by a physics-based model simulating a dipole-dipole interaction. A new extended local coherent interaction is introduced defined in terms of extended structure tensor of the image to give priority to parallel coherence vectors.

Stent implantation for coronary disease treatment is a highly important minimally invasive technique that avoids surgery interventions. In order to assure the success of such an intervention, it is very important to determine the real length of the lesion as exactly as possible. Currently, lesion measures are performed directly from the angiography without considering the system projective parameters or, alternatively, from the 3D reconstruction obtained from a correspondence of points defined by the physicians. In this paper, we present a method for 3D vessel reconstruction from biplane images by means of deformable models. In particular, we study the known shortcoming of point-based 3D vessel reconstruction (no intersection of projective beams) and illustrate that by using snakes the reconstruction error is minimal. We validate out method by a computer-generated phantom, a real phantom and coronary vessels.

We address the object recognition problem in a probabilistic framework to detect and describe object appearance through image features organized by means of active contour models. We consider the formulation of saliency in terms of visual similarity embedded in the probabilistic principal component analysis framework. A likelihood of object structure detection is obtained using the relation between the visual field and the internal object representation. Deformable models are employed introducing a computational methodology for a perceptual organisation of image features as an abstract understanding of the integration between structure and constraints of the visual information-processing problem. A specific application of the integrated approach for vessels segmentation in angiography is considered and the results are encouraging.

We propose a new statistic deformable model that we call discriminant snake for 3D reconstruction in volumetric images. Our discriminant snake generalises the classical snake attracted by edge points; it deforms due to a generalised contour representation. The snake selects and classifies image features by a parametric classifier and each snaxel deforms to minimise the dissimilarity between the learned and found image features inside the feature space. We apply our statistic snake to segment anatomical organs and the results are very encouraging.

We introduce a new deformable model, called eigensnake, for segmentation of elongated structures in a probabilistic framework. Instead of snake attraction by specific image features extracted independently of the snake, our eigensnake learns an optimal object description and searches for such image feature in the target image. This is achieved applying principal component analysis on image responses of a bank of Gaussian derivative filters. Therefore, attraction by eigensnakes is defined in terms of classification of image features. The potential energy for the snake is defined in terms of likelihood in the feature space and incorporated into a new energy minimising scheme. Hence, the snake deforms to minimise the mahalanobis distance in the feature space. A real application of segmenting and tracking coronary vessels in angiography is considered and the results are very encouraging.

Traditional techniques for tracking nonrigid objects such as optical flow, correlation, active contours or color, cannot deal with situations where image changes are not due to motion but appearance (e.g. tracking the lips when the teeth appear). Two main contributions for appearance tracking of flexible objects are proposed. The first one is a flexible generalization of eigentracking within the same robust continuous optimization framework. The second one is a generalization of traditional graylevel eigenspaces, constructing a multiple channel "eigenspace" using filter responses to give robustness against variations in the training conditions such as illumination changes. Additionally, 3D geometric transformations are incorporated, a regularization term is added for numerical stability reasons and the optimization problem is solved in closed form. Experiments on lip tracking are reported.

The problem of segmenting agricultural fields in aerial images is still a manual work in most geographic information system requiring repetitive, tedious and time-consuming human work. Here, we address the problem of semiautomatic segmenting agricultural fields by region competition technique that integrates region growing and deformable models. The deformable model dynamically adapts its contour, analyzing homogeneous parcels in an energy-minimizing framework. To assure the optimal image segmentation and practical applicability of the approach, we study different aspects: parameterization, convergence criteria and user interaction. The successful results obtained have allowed introducing the region competition technique in a teledetection environment.

This paper provides a generalization of non-reducible descriptors. Non-reducible descriptors are used in supervised pattern recognition problems when the patterns descriptions consist of Boolean variables. This generalization extends the concept of distance between patterns of different classes. A mathematical model to construct generalized non-reducible descriptors, a computational procedure, and numerical examples are discussed.

In this invited paper, an overview of techniques developed at the Cardiovascular Image Analysis Laboratory at Washington University is discussed. At the core of the authors' methodologies lie flexible shape models, which are employed in automated as well as semi-automated analysis of cardiac MRI and X-ray angiography images. The mathematical bases used for the flexible templates are of the B-spline variety, providing compact representation and interactive capabilities for manipulation of curves, surfaces, and volumes.

We consider the classical pattern recognition problem. A model of construction of Boolean decision rules is implemented. Computational procedures for the construction of nonreducible descriptors and clusters are briefly discussed. Applications of nonreducible descriptors to ECG analysis and ECG recognition are suggested.

Despite the promising results of numerous applications, the hitherto proposed snake techniques share some common problems: snake attraction by spurious edge points, snake degeneration (shrinking and flattening), convergence and stability of the deformation process, snake initialization and local determination of the parameters of elasticity. We argue here that these problems can be solved only when all the snake aspects are considered. The snakes proposed here implement a new potential field and external force in order to provide a deformation convergence, attraction by both near and far edges as well as snake behaviour selective according to the edge orientation. Furthermore, we conclude that in the case of model-based segmentation, the internal force should include structural information about the expected snake shape. Experiments using this kind of snakes for segmenting bones in complex hand radiographs show a significant improvement.>

A method of solving supervised pattern recognition problems based on the model of learning Boolean formulas is suggested. It is proved that this method of learning is of NP-complexity. An efficient learning procedure using some tools of combinatorics and graph theory is proposed. The suggested method differs from those known in its diminished number of computational operations. The results obtained are applied to supervised image recognition problems.>