This work tackles the automatic prediction of grasping intention of humans observing their environment. Our target application is the assistance to people with motor disabilities and potential cognitive impairments, using assistive robotics. Our proposal leverages the analysis of human attention captured in the form of gaze fixations recorded by an eye-tracker on the first person video, as the anticipation of prehension actions is a well studied and well known phenomenon. We propose a multi-task system that simultaneously addresses the prediction of human attention in the near future, and the anticipation of grasping actions. Visual attention is modeled as a competitive process between a discrete set of states, each one associated to a well-known gaze movement pattern from visual psychology. We additionally consider an asymmetric multi-task problem, where attention modeling is an auxiliary task that helps to regularize the learning process of the main action prediction task, and propose a constrained multi-task loss that naturally deals with this asymmetry. Our model shows superior performance than other losses for dynamic multi-task learning, current dominant deep architectures for general action forecasting and particularly-tailored models for predicting grasping intention. In particular, it provides state-of-the-art performance in three datasets for egocentric action anticipation, with an average precision of 0.569 and 0.524 in GITW and Sharon datasets, respectively, and an accuracy of 89.2% and a success rate of 51.7% in Invisible dataset.

Cell detection and tracking applied to in vivo fluorescence microscopy has become an essential tool in biomedicine to characterize 4D (3D space plus time) biological processes at the cellular level. Traditional approaches to cell motion analysis by microscopy imaging, although based on automatic frameworks, still require manual supervision at some points of the system. Hence, when dealing with a large amount of data, the analysis becomes incredibly time-consuming and typically yields poor biological information. In this paper, we propose a fully-automated system for segmentation, tracking and feature extraction of migrating cells within blood vessels in 4D microscopy imaging. Our system consists of a robust 3D convolutional neural network (CNN) for joint blood vessel and cell segmentation, a 3D tracking module with collision handling, and a novel method for feature extraction, which takes into account the particular geometry in the cell-vessel arrangement. Experiments on a large 4D intravital microscopy dataset show that the proposed system achieves a significantly better performance than the state-of-the-art tools for cell segmentation and tracking. Furthermore, we have designed an analytical method of cell behaviors based on the automatically extracted features, which supports the hypotheses related to leukocyte migration posed by expert biologists. This is the first time that such a comprehensive automatic analysis of immune cell migration has been performed, where the total population under study reaches hundreds of neutrophils and thousands of time instances.

Modern computer vision techniques have to deal with vast amounts of visual data, which implies a computational effort that has often to be accomplished in broad and challenging scenarios. The interest in efficiently solving these image and video applications has led researchers to develop methods to expertly drive the corresponding processing to conspicuous regions that either depend on the context or are based on specific requirements. In this paper, we propose a general hierarchical probabilistic framework, independent of the application scenario, and relied on the most outstanding psychological studies about attention and eye movements which support that guidance is not based directly on the information provided by early visual processes but on a contextual representation that arose from them. The approach defines the task of context-driven visual attention as a mixture of latent sub-tasks, which are, in turn, modeled as a combination of specific distributions associated to low-, mid-, and high-level spatio-temporal features. Learning from fixations gathered from human observers, we incorporate an intermediate level between feature extraction and visual attention estimation that enables to obtain comprehensively guiding representations. The experiments show how our proposal successfully learns particularly adapted hierarchical explanations of visual attention in diverse video genres, outperforming several leading models in the literature.

Traditional approaches to automatic diagnosis of skin lesions consisted of classifiers working on sets of hand-crafted features, some of which modeled lesion aspects of special importance for dermatologists. Recently, the broad adoption of convolutional neural networks (CNNs) in most computer vision tasks has brought about a great leap forward in terms of performance. Nevertheless, with this performance leap, the CNN-based computer-aided diagnosis (CAD) systems have also brought a notable reduction of the useful insights provided by hand-crafted features. This paper presents DermaKNet, a CAD system based on CNNs that incorporates specific subsystems modeling properties of skin lesions that are of special interest to dermatologists aiming to improve the interpretability of its diagnosis. Our results prove that the incorporation of these subsystems not only improves the performance, but also enhances the diagnosis by providing more interpretable outputs.

License plate detection is a common problem in traffic surveillance applications. Although some solutions have been proposed in the literature, their success is usually restricted to very specific scenarios, with their performance dropping in more demanding conditions. One of the main challenges to be addressed for this kind of systems is the varying scale of the license plates, which depends on the distance between the vehicles and the camera. Traditionally, systems have handled this issue by sequentially running single-scale detectors over a pyramid of images. This approach, although simplifies the training process, requires as many evaluations as considered scales, which leads to running times that grow linearly with the number of scales considered. In this paper, we propose a scale-adaptive deformable part-based model which, based on a well-known boosting algorithm, automatically models scale during the training phase by selecting the most prominent features at each scale and notably reduces the test detection time by avoiding the evaluation at different scales. In addition, our method incorporates an empirically constrained-deformation model that adapts to different levels of deformation shown by distinct local features within license plates. As shown in the experimental section, the proposed detector is robust and scale and perspective independent and can work in quite diverse scenarios. Experiments on two datasets show that the proposed method achieves a significantly better performance in comparison with other methods of the state of the art.

Content-based image representation is a very challenging task if we restrict to their visual content. However, associated metadata (such as tags or geolocation) become a valuable source of complementary information that may help to enhance the current system performance. In this paper, we propose an automatic training framework that uses both image visual contents and metadata to fine tune deep Convolutional Neural Networks (CNNs), providing better image descriptors adapted to certain locations, such as cities or regions. Specifically, we propose to estimate some weak labels by combining visual- and location-related information and incorporate them to a novel loss-function over pairs of images. Our experiments on a landmark discovery task show that this novel training procedure enhances the performance up to a 55% over well-established CNN-based models and is free from overfitting.

Incorporating user perception into visual content search and understanding tasks has become one of the major trends in multimedia retrieval. We tackle the problem of object recognition guided by user perception, as indicated by his gaze during visual exploration, in the application domain of assistance to upper-limb amputees. Although selecting the object to be grasped represents a task-driven visual search, human gaze recordings are noisy due to several physiological factors. Hence, since gaze does not always point to the object of interest, we use video-level weak annotations indicating the object to be grasped, and propose a video-level weak loss in classification with Deep CNNs. Our results show that the method achieves notably better performance than other approaches over a complex real-life dataset specifically recorded, with optimal performance for fixation times around 400-800ms, producing a minimal impact on subjects' behavior.

In the last few years, large-scale image retrieval has attracted a lot of attention from the multimedia community. Usual approaches addressing this task first generate an initial ranking of the reference images using fast approximations that do not take into consideration the spatial arrangement of local features in the image (e.g., the bag-of-words paradigm). The top positions of the rankings are then re-estimated with verification methods that deal with more complex information, such as the geometric layout of the image. This verification step allows pruning of many false positives at the expense of an increase in the computational complexity, which may prevent its application to large-scale retrieval problems. This paper describes a geometric method known as neighborhood matching (NM), which revisits the keypoint matching process by considering a neighborhood around each keypoint and improves the efficiency of a geometric verification step in the image search system. Multiple strategies are proposed and compared to incorporate NM into a large-scale image retrieval framework. A detailed analysis and comparison of these strategies and baseline methods have been investigated. The experiments show that the proposed method not only improves the computational efficiency, but also increases the retrieval performance and outperforms state-of-the-art methods in standard datasets, such as the Oxford 5 k and 105 k datasets, for which the spatial verification step has a significant impact on the system performance.

Behavioural analysis in instrumental activities of daily living has become a powerful tool in clinical studies and rises the question of what objects are manipulated by patients. In this paper we present a top-down probabilistic visual attention model for manipulated object recognition in egocentric video content. Although arms often occlude objects and are usually seen as a burden for many vision systems, they become an asset in our approach, as we extract both global and local features describing their geometric layout and pose, as well as the objects being manipulated. We integrate this information in a probabilistic generative model, provide update equations that automatically compute the model parameters optimizing the likelihood of the data, and design a method to generate maps of visual attention that are later used in an object-recognition framework. This task-driven assessment reveals that the proposed method outperforms the state of the art in object recognition for egocentric video content.

This paper tackles the problem of automatic brain tumor classification from Magnetic Resonance Imaging (MRI) where, traditionally, general-purpose texture and shape features extracted from the Region of Interest (tumor) have become the usual parameterization of the problem. Two main contributions are made in this context. First, a novel set of clinical-based features that intend to model intuitions and expert knowledge of physicians is suggested. Second, a system is proposed that is able to fuse multiple individual scores (based on a particular MRI sequence and a pathological indicator present in that sequence) by using a Bayesian model that produces a global system decision. This approximation provides a quite flexible solution able to handle missing data, which becomes a very likely case in a realistic scenario where the number clinical tests varies from one patient to another. Furthermore, the Bayesian model provides extra information concerning the uncertainty of the final decision. Our experimental results prove that the use of clinical-based feature leads to a significant increment of performance in terms of Area Under the Curve (AUC) when compared to a state-of-the art reference. Furthermore, the proposed Bayesian fusion model clearly outperforms other fusion schemes, especially when few diagnostic tests are available.

This paper proposes a probabilistic generative model that concurrently tackles the problems of image retrieval and region-of-interest (ROI) segmentation. Specifically, the proposed model takes into account several properties of the matching process between two objects in different images, namely: objects undergoing a geometric transformation, typical spatial location of the region of interest, and visual similarity. In this manner, our approach improves the reliability of detected true matches between any pair of images. Furthermore, by taking advantage of the links to the ROI provided by the true matches, the proposed method is able to perform a suitable ROI segmentation. Finally, the proposed method is able to work when there is more than one ROI in the query image. Our experiments on two challenging image retrieval datasets proved that our approach clearly outperforms the most prevalent approach for geometrically constrained matching and compares favorably to most of the state-of-the-art methods. Furthermore, the proposed technique concurrently provided very good segmentations of the ROI. Furthermore, the capability of the proposed method to take into account several objects-of-interest was also tested on three experiments: two of them concerning image segmentation and object detection in multi-object image retrieval tasks, and another concerning multiview image retrieval. These experiments proved the ability of our approach to handle scenarios in which more than one object of interest is present in the query.

In this paper we propose a system for automatic detection of specific events and abnormal behaviors in crowded scenes. In particular, we focus on the parametrization by proposing a set of mid-level spatio-temporal features that successfully model the characteristic motion of typical events in crowd behaviors. Furthermore, due to the fact that some features are more suitable than others to model specific events of interest, we also present an automatic process for feature selection. Our experiments prove that the suggested feature set works successfully for both explicit event detection and distance-based anomaly detection tasks. The results on PETS for explicit event detection are generally better than those previously reported. Regarding anomaly detection, the proposed method performance is comparable to those of state-of-the-art method for PETS and substantially better than that reported for Web dataset.

For the last few years bag-of-words models have been succesfully applied to the information retrieval field. However their application to visual content suffers from an important shortcoming: they model images as sets of unordered visual words rather than consider their spatial and geometric layout. Visual information is highly organized along the dimensions of an image and algorithms should make use of this to enhance the performance of several visual processing tasks. In this paper, a generative model is proposed that fuses both the local information obtained from visual words and the global geometric layout given by a previous segmentation of the image. Furthermore, the model considers inter-region influences so topics can spread along the image and, thus, generate final segmentations in which regions represent semantic concepts. The proposed model is succesfully tested on three different tasks.

Segmentation algorithms traditionally employ low-level features to divide images into different regions that show a certain degree of homogeneity. However, low-level features, spatial or temporal, are not always reliable when processing real-world video sequences, because of issues like illuminations or complex backgrounds. Furthermore, real world objects can be composed of different regions with heterogeneous features. Although the inclusion of motion can mitigate some of these effects, many problems are still present. This paper proposes the utilization of some spatio-temporal mid-level features that are related, on the one hand, to geometric properties of real objects and, on the other, to well-known motion patterns. Specifically, the proposed algorithm uses a mid-level module that controls the subsequent segmentation using these kinds of features. Some experiments and evaluations show that the inclusion of mid-level features can help to obtain perceptually more meaningful segmentations, thus resulting in regions that are closer to semantic concepts.

In most video coding standards, motion estimation becomes the most time-consuming subsystem. Consequently, in the last few years, a great deal of effort has been devoted to the research of novel algorithms capable of saving computations with minimal effects on the coding quality. Adaptive algorithms and particularly multipattern solutions, have evolved as the most robust general-purpose solutions owing to two main reasons: 1) real video sequences usually exhibit a wide-range of motion content, from uniform to random, and 2) a vast amount of coding applications have appeared demanding different degrees of coding quality. In this study, we propose an adaptive algorithm, called motion classification-based search (MCS), which makes use of an especially tailored classifier that detects some motion cues and chooses the search pattern that best fits them. The MCS has been experimentally assessed for a comprehensive set of selected video sequences and qualities. Our experimental results show that MCS notably reduces the computational cost up to 55% and 84% in search points, with respect to two state-of-the-art methods, while maintaining the quality.

Motion estimation is the most time-consuming subsystem in a video codec. Thus, more efficient methods of motion estimation should be investigated. Real video sequences usually exhibit a wide-range of motion content as well as different degrees of detail, which become particularly difficult to manage by typical block-matching algorithms. Recent developments in the area of motion estimation have focused on the adaptation to video contents. Adaptive thresholds and multi-pattern search algorithms have shown to achieve good performance when they success to adjust to motion characteristics. This paper proposes an adaptive algorithm, called MCS, that makes use of an especially tailored classifier that detects some motion cues and chooses the search pattern that best fits to them. Specifically, a hierarchical structure of binary linear classifiers is proposed. Our experimental results show that MCS notably reduces the computational cost with respect to an state-of-the-art method while maintaining the quality.

A rate control (RC) algorithm for H.264 video coding with stored-B (SB) pictures is proposed for low-delay applications. Different models for P and SB pictures are defined for a better QP and MAD estimation. Furthermore, a novel saw-tooth shaped model of target buffer level has also been introduced for a proper bit allocation in GOP structures with SB pictures. Our experimental results show that this proposal outperforms the reference software RC in terms of buffer occupancy and target bit rate adjustment at the expense of slight quality reduction.

In several video coding standards, such as H.264, motion estimation becomes the most-time consuming subsystem. Therefore, recently research on video coding focuses on the development of novel algorithms able to save computations with minimal effects over the coding distortion. Due to the fact that real video sequences usually exhibit a wide-range of motion content, from uniform to random, and to the vast amount of coding applications demanding different degrees of coding quality, adaptive algorithms have revealed as the most robust general purpose solutions. In particular, multi-pattern algorithms can adapt to video contents as well as to required coding quality by means of the use of a set of heterogeneus search patterns, each one adapting better to particular motion and quality requirements. This paper applies some improvements to the Motion Classification based Search, an adaptive multi-pattern algorithm based on motion classification techniques. Our experimental results show that MCS notably reduces the computational cost with respect to some well-known algorithms while maintaining the quality.