We propose an unsupervised probabilistic framework for learning a human-centred representation of a person’s environment from first-person video. Specifically, non-geometric maps modelled as hierarchies of probabilistic place graphs and view graphs are learned. Place graphs model a user’s patterns of transition between physical locations whereas view graphs capture an aspect of user behaviour within those locations. Furthermore, we describe an implementation in which the notion of place is divided into stations and the routes that interconnect them. Stations typically correspond to rooms or areas where a user spends time. Visits to stations are temporally segmented based on qualitative visual motion. We describe how to learn maps online in an unsupervised manner, and how to localise the user within these maps. We report experiments on two datasets, including comparison of performance with and without view graphs, and demonstrate better online mapping than when using offline clustering.

Graph learning methods have achieved noteworthy performance in disease diagnosis due to their ability to represent unstructured information such as inter-subject relationships. While it has been shown that imaging, genetic and clinical data are crucial for degenerative disease diagnosis, existing methods rarely consider how best to use their relationships. How best to utilize information from imaging, genetic and clinical data remains a challenging problem. This study proposes a novel graph-based fusion (GBF) approach to meet this challenge. To extract effective imaging-genetic features, we propose an imaging-genetic fusion module which uses an attention mechanism to obtain modality-specific and joint representations within and between imaging and genetic data. Then, considering the effectiveness of clinical information for diagnosing degenerative diseases, we propose a multi-graph fusion module to further fuse imaging-genetic and clinical features, which adopts a learnable graph construction strategy and a graph ensemble method. Experimental results on two benchmarks for degenerative disease diagnosis (Alzheimers Disease Neuroimaging Initiative and Parkinson's Progression Markers Initiative) demonstrate its effectiveness compared to state-of-the-art graph-based methods. Our findings should help guide further development of graph-based models for dealing with imaging, genetic and clinical data.

Deep convolutional neural networks are widely used to learn feature spaces for image classification tasks. We propose cam-softmax, a generalisation of the final layer activations and softmax function, that encourages deep feature spaces to exhibit high intra-class compactness and high inter-class separability. We provide an algorithm to automatically adapt the method's main hyperparameter so that it gradually diverges from the standard activations and softmax method during training. We report experiments using CASIA-Webface, LFW, and YTF face datasets demonstrating that cam-softmax leads to representations well suited to open-set face recognition and face pair matching. Furthermore, we provide empirical evidence that cam-softmax provides some robustness to class labelling errors in training data, making it of potential use for deep learning from large datasets with poorly verified labels.

Several advantages would arise from the automated detection of pathologies of pig carcasses, including avoidance of the inherent risks of subjectivity and variability between human observers. Here, we develop a novel automated classification of two porcine offal pathologies at abattoir: a focal, localized pathology of the liver and a diffuse pathology of the heart, as cases in point. We develop a pattern recognition system based on machine learning to identify those organs that exhibit signs of the pathology of interest. Specifically, deep neural networks are trained to produce probability heat maps, highlighting regions on the surface of an organ potentially affected by a given condition. A final classification stage then decides whether a given organ is affected by the condition in question based on statistics computed from the heat map. We compare outcomes of automated classification with classification by expert pathologists. Results show the classification of liver and heart pathologies in agreement with an expert at levels comparable to, or exceeding, interexpert agreement. A system using methods such as those presented here has potential to overcome the limitations of human-based abattoir inspection, especially if this is based on visual-only inspection, and ultimately to provide a new gold standard for pathology.

Activities such as those involved in food preparation involve interactions between hands, tools and multiple manipulated objects that affect them in visually complex ways making recognition of their constituent actions challenging. We describe a system that classifies action classes in such a setting based on discriminative spatio-temporal superpixel groups. The entire system operates sequentially enabling online action recognition. We obtain state-of-the-art results whilst employing a compact, interpretable representation.

Extensions to auto-context segmentation are proposed and applied to segmentation of multiple organs in porcine offal as a component of an envisaged system for post-mortem inspection at abbatoir. In common with multi-part segmentation of many biological objects, challenges include variations in configuration, orientation, shape, and appearance, as well as inter-part occlusion and missing parts. Auto-context uses context information about inferred class labels and can be effective in such settings. Whereas auto-context uses a fixed prior atlas, we describe an adaptive atlas method better suited to represent the multimodal distribution of segmentation maps. We also design integral context features to enhance context representation. These methods are evaluated on a dataset captured at abbatoir and compared to a method based on conditional random fields. Results demonstrate the appropriateness of auto-context and the beneficial effects of the proposed extensions for this application.

We present a novel method to segment instances of glandular structures from colon histopathology images. We use a structure learning approach which represents local spatial configurations of class labels, capturing structural information normally ignored by sliding-window methods. This allows us to reveal different spatial structures of pixel labels (e.g., locations between adjacent glands, or far from glands), and to identify correctly neighboring glandular structures as separate instances. Exemplars of label structures are obtained via clustering and used to train support vector machine classifiers. The label structures predicted are then combined and post-processed to obtain segmentation maps. We combine hand-crafted, multi-scale image features with features computed by a deep convolutional network trained to map images to segmentation maps. We evaluate the proposed method on the public domain GlaS data set, which allows extensive comparisons with recent, alternative methods. Using the GlaS contest protocol, our method achieves the overall best performance.

One approach to the recognition of complex human activities is to use feature descriptors that encode visual interactions by describing properties of local visual features with respect to trajectories of tracked objects. We explore an example of such an approach in which dense tracklets are described relative to multiple reference trajectories, providing a rich representation of complex interactions between objects of which only a subset can be tracked. Specifically, we report experiments in which reference trajectories are provided by tracking inertial sensors in a food preparation scenario. Additionally, we provide baseline results for HOG, HOF and MBH, and combine these features with others for multi-modal recognition. The proposed histograms of relative tracklets (RETLETS) showed better activity recognition performance than dense tracklets, HOG, HOF, MBH, or their combination. Our comparative evaluation of features from accelerometers and video highlighted a performance gap between visual and accelerometer-based motion features and showed a substantial performance gain when combining features from these sensor modalities. A considerable further performance gain was observed in combination with RETLETS and reference tracklet features.

Difficulties can arise from the segmentation of three-dimensional objects formed by multiple non-rigid parts represented in two-dimensional images. Problems involving parts whose spatial arrangement is subject to weak restrictions, and whose appearance and form change across images, can be particularly challenging. Segmentation methods that take into account spatial context information have addressed these types of problem, which often involve image data of a multi-modal nature. An attractive feature of the auto-context (AC) technique is that a prior "atlas", typically obtained by averaging multiple label maps created by experts, can be used as an initial source of contextual data. However, a prior obtained in this way is likely to hide the inherent multi-modality of the data. We propose a modification of AC in which a probabilistic atlas of part locations is iteratively improved and made available as an additional source of information. We illustrate this technique with the problem of segmenting individual organs in images of pig offal, reporting statistically significant improvements in relation to both conventional AC and a state-of-the-art technique based on conditional random fields.

Immunofluorescence antinuclear antibody tests are important for diagnosis and management of autoimmune conditions; a key step that would benefit from reliable automation is the recognition of subcellular patterns suggestive of different diseases. We present a system to recognize such patterns, at cellular and specimen levels, in images of HEp-2 cells. Ensembles of SVMs were trained to classify cells into six classes based on sparse encoding of texture features with cell pyramids, capturing spatial, multi-scale structure. A similar approach was used to classify specimens into seven classes. Software implementations were submitted to an international contest hosted by ICPR 2014 (Performance Evaluation of Indirect Immunofluorescence Image Analysis Systems). Mean class accuracies obtained on heldout test data sets were 87.1% and 88.5% for cell and specimen classification respectively. These were the highest achieved in the competition, suggesting that our methods are state-of-the-art. We provide detailed descriptions and extensive experiments with various features and encoding methods.

Content‐based image retrieval (CBIR) systems often incorporate a relevance feedback mechanism in which retrieval is adapted based on users identifying images as relevant or irrelevant. Such relevance decisions are often assumed to be category‐based. However, forcing a user to decide upon category membership of an image, even when unfamiliar with a database and irrespective of context, is restrictive. An alternative is to obtain user feedback in the form of relative similarity judgments. The ability of a user to provide meaningful feedback depends on the interface that displays retrieved images and facilitates the feedback. Similarity‐based 2D layouts provide context and can enable more efficient visual search. Motivated by these observations, this study describes and evaluates an interactive image browsing and retrieval approach based on relative similarity feedback obtained from 2D image layouts. It incorporates online maximal‐margin learning to adapt the image similarity metric used to perform retrieval. A user starts a session by browsing a collection of images displayed in a 2D layout. He/she may choose a query image perceived to be similar to the envisioned target image. A set of images similar to the query are then returned. The user can then provide relational feedback and/or update the query image to obtain a new set of images. Algorithms for CBIR are often characterised empirically by simulating usage based on pre‐defined, fixed category labels, deeming retrieved results as relevant if they share a category label with the query. In contrast, the purpose of the system in this study is to enable browsing and retrieval without predefined categories. Therefore evaluation is performed in a target‐based setting by quantifying the efficiency with which target images are retrieved given initial queries.

This paper introduces a publicly available dataset of complex activities that involve manipulative gestures. The dataset captures people preparing mixed salads and contains more than 4.5 hours of accelerometer and RGB-D video data, detailed annotations, and an evaluation protocol for comparison of activity recognition algorithms. Providing baseline results for one possible activity recognition task, this paper further investigates modality fusion methods at different stages of the recognition pipeline: (i) prior to feature extraction through accelerometer localization, (ii) at feature level via feature concatenation, and (iii) at classification level by combining classifier outputs. Empirical evaluation shows that fusing information captured by these sensor types can considerably improve recognition performance.

We present RERBEE (robust efficient registration via bifurcations and elongated elements), a novel feature-based registration algorithm able to correct local deformations in high-resolution ultra-wide field-of-view (UWFV) fluorescein angiogram (FA) sequences of the retina. The algorithm is able to cope with peripheral blurring, severe occlusions, presence of retinal pathologies and the change of image content due to the perfusion of the fluorescein dye in time. We have used the computational power of a graphics processor to increase the performance of the most computationally expensive parts of the algorithm by a factor of over × 1300, enabling the algorithm to register a pair of 3900 × 3072 UWFV FA images in 5-10 min instead of the 5-7 h required using only the CPU. We demonstrate accurate results on real data with 267 image pairs from a total of 277 (96.4%) graded as correctly registered by a clinician and 10 (3.6%) graded as correctly registered with minor errors but usable for clinical purposes. Quantitative comparison with state-of-the-art intensity-based and feature-based registration methods using synthetic data is also reported. We also show some potential usage of a correctly aligned sequence for vein/artery discrimination and automatic lesion detection.

Markov random field pixel labelling is often used to obtain image segmentations in which each segment or region is labelled according to its attributes such as colour or texture. This paper explores the use of such a representation for image classification. In particular, the problem of classifying textile images according to design type is addressed. Regions with the same label are treated as a group and each group is associated uniquely with a vertex in an undirected, weighted graph. Each region group is represented as a bag of shape descriptors. Edges in the graph denote either the extent to which the groups’ regions are spatially adjacent or the dissimilarity of their respective bags of shapes. Series of unweighted graphs are obtained by removing edges in order of weight. Finally, an image is represented using its shape descriptors along with features derived from the chromatic numbers or domination numbers of the unweighted graphs and their complements. Experimental results are reported on a challenging classification task using images from a textile design archive

The use of region shape descriptors was investigated for categorisation of textile design images. Images were segmented using MRF pixel labelling and the shapes of regions obtained were described with generic Fourier descriptors. Each image was represented as a bag of shapes. A simple yet competitive classification scheme based on nearest neighbour class-based matching was used. Classification performance was compared to that obtained when using bags of SIFT features.

A method for Gaussian process learning of a scalar function from a set of pair-wise order relationships is presented. Expectation propagation is used to obtain an approximation to the log marginal likelihood which is optimised using an analytical expression for its gradient. Experimental results show that the proposed method performs well compared with a previous method for Gaussian process preference learning.

Mechanisms for visualizing image collections are essential for browsing and exploring their content. This is especially true when metadata are ineffective in retrieving items due to the sparsity or esoteric nature of text. An obvious approach is to automatically lay out sets of images in ways that reflect relationships between the items. However, dimensionality reduction methods that map from high-dimensional content-based feature distributions to low-dimensional layout spaces for visualization often result in displays in which many items are occluded whilst large regions are empty or only sparsely populated. Furthermore, such methods do not consider the shape of the region of layout space to be populated. This paper proposes a method, high-entropy layout distributions. that addresses these limitations. Layout distributions with low differential entropy are penalized. An optimization strategy is presented that finds layouts that have high differential entropy and that reflect inter-image similarities. Efficient optimization is obtained using a step-size constraint and an approximation to quadratic (Renyi) entropy. Two image archives of cultural and commercial importance are used to illustrate and evaluate the method. A comparison with related methods demonstrates its effectiveness.

An approach to the analysis of images of regular texture is proposed in which lattice hypotheses are used to define statistical models. These models are then compared in terms of their ability to explain the image. A method based on this approach is described in which lattice hypotheses are generated using analysis of peaks in the image autocorrelation function, statistical models are based on Gaussian or Gaussian mixture clusters, and model comparison is performed using the marginal likelihood as approximated by the Bayes Information Criterion (BIC). Experiments on public domain regular texture images and a commercial textile image archive demonstrate substantially improved accuracy compared to two competing methods. The method is also used for classification of texture images as regular or irregular. An application to thumbnail image extraction is discussed. These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Statistical shape models are often learned from examples based on landmark correspondences between annotated examples. A method is proposed for learning such models from contours with inconsistent bifurcations and loops. Automatic segmentation of tibial and femoral contours in knee X-ray images is investigated as a step towards reliable, quantitative radiographic analysis of osteoarthritis for diagnosis and assessment of progression. Results are presented using various features, the Mahalanobis distance, distance weighted K-nearest neighbours, and two relevance vector machine-based methods as quality of fit measure.

A probabilistic method is proposed for segmentation of multiple objects that overlap or are in close proximity to one another. A likelihood function is formulated that explicitly models overlapping object appearance. Priors on global appearance and geometry (including shape) are learned from example images. Markov chain Monte Carlo methods are used to obtain samples from a posterior distribution over model parameters from which expectations can be estimated. The method is described in detail for the problem of segmenting femur and tibia in x-ray images. The result is a probabilistic segmentation that quantifies uncertainty so that measurements such as joint space can be made with associated uncertainty.

www.computing.dundee.ac.uk/projects/Vision Statistical shape models are often learned from examples based on landmark correspondences between annotated examples. A method is proposed for learning such models from contours with inconsistent bifurcations and loops. It is evaluated on the task of segmenting tibial contours in knee radiographs. Results are presented using various features, distance weighted K−nearest neighbours and differing eigenspace shape constraints. 1

A likelihood formulation for human tracking is presented based upon matching feature statistics on the surface of an articulated 3D model. A benefit of such a formulation over current techniques is that it provides a dense, object-based cue. Multi-dimensional histograms are used to represent the feature distributions. Different histogram similarity measures are evaluated for tracking purposes. An on-line region grouping algorithm, driven by prior knowledge of clothing structure, is derived that improves the appearance estimation and computational efficiency. Finally, we demonstrate that the smooth, broad likelihood response allows efficient inference using coarse sampling and local optimisation. Results from tracking real world sequences are presented.

A computer vision system for tracking multiple people in relatively unconstrained environments is described. Tracking is performed at three levels of abstraction: regions, people and groups. A novel, adaptive background subtraction method that combines colour and gradient information is used to cope with shadows and unreliable colour cues. People are tracked through mutual occlusions as they form groups and part from one another. Strong use is made of colour information to disambiguate occlusions and to provide qualitative estimates of depth ordering and position during occlusion. Some simple interactions with objects can also be detected. The system is tested using indoor and outdoor sequences. It is robust and should provide a useful mechanism for bootstrapping and reinitialisation of tracking using more-specific but less-robust human models.

Analysis has shown that a proportion of motor vehicle accidents are caused by driver inattention induced by the operation of secondary controls, e.g. radio and climate controls. We have developed a novel, prototype, finger-pointing interface for operating such controls. The current physical controls are replaced by a computer interface, which has the advantage of being situated much closer to the driver's normal line of sight. This interface is operated by the driver using pointing gestures that will ultimately be identified using computer vision. The potential safety benefits that may accrue from this interface are outlined. The prototype interface currently uses an electromagnetic sensor to determine the direction of the pointing gesture. This paper describes this prototype and reports preliminary results from use of the graphical user interface.

The paper describes a computer vision system for tracking a driver's hands. The applications of such a system are discussed with particular attention given to the control of a novel vehicle systems interface. This interface is controlled by pointing at the required function on a display mounted behind the top of the steering wheel. The vision system must detect and track the driver's hands to allow efficient recovery of a search space in which to look for a pointing finger. Cameras mounted above the driver provide images of the steering wheel area. The scene is segmented using adaptive background and foreground models. A distance transform is applied to the resulting contours. A geometric model is fitted to the resulting distance map by maximising an objective function over the model parameters in a local search space defined by the output of a Kalman filter. The appearance-based model fitting process can resolve ambiguous situations such as the hands touching or crossing, and parting again. The system is demonstrated running at 10 Hz on a standard PC.

We present a method for learning appearance models that can be used to recognise and track both 3D head pose and identities of novel subjects with continuous head movement across the view-sphere. We describe an automatic face data acquisition system based on a magnetic sensor and a calibrated camera. The system enabled us to obtain systematically a database of face images with labelled 3D poses across a view-sphere of \\Sigma90 ffi yaw and \\Sigma30 ffi tilt at intervals of 10 ffi . The database was used to learn appearance models of unseen faces based on similarity measures to prototype faces. The method is computationally efficient and enables real-time performance with ease. 1 Introduction To be able to recognise faces of moving people not only requires the ability to label novel face images with known identities, but also needs detecting and tracking of faces over time [1]. We refer to this as the task of associating faces. We adopt the view such a task can be better achieved...

An approach to gesture recognition is presented in which gestures are modelled probabilistically as sequences of visual events. These events are matched to visual input using probabilistic models estimated from motion feature trajectories. The features used are motion image moments. The method was applied to a set of gestures defined within the context of an application in visually mediated interaction in which they would be used to control an active teleconferencing camera. The approach is computationally efficient, allowing real-time performance to be obtained. 1

We describe a dynamic face tracking system based on an integrated motion-based object tracking and model-based face detection framework. The motion-based tracker focuses attention for the face detector whilst the latter aids the tracking process. The system produces segmented face sequences from complex scenes with poor viewing conditions in surveillance applications. We also investigate a Gabor wavelet transform as a representation scheme for capturing head rotations in depth. Principal components analysis was used to visualise the manifolds described by pose changes. Qualitative results are given. 1 Introduction In order to analyse and recognise peoples' faces in realistically unconstrained environments, robust tracking and segmentation are needed to provide sequences of normalised face images. Although such a normalisation process is often treated as a separate preprocessing step, it is an inherent part of face recognition. The ability of a system to produce normalised face sequenc...

Visual perception of faces is invariant under many transformations, perhaps the most problematic of which is pose change (face rotating in depth). We use a variation of Gabor wavelet transform (GWT) as a representation framework for investigating face pose measurement. Dimensionality reduction using principal components analysis (PCA) enables pose changes to be visualised as manifolds in low-dimensional subspaces and provides a useful mechanism for investigating these changes. The effectiveness of measuring face pose with GWT representations was examined using PCA. We discuss our experimental results and draw a few preliminary conclusions.

Robust tracking and segmentation of faces is a prerequisite for face analysis and recognition. In this paper we describe an approach to this problem which is well suited to surveillance applications with poorly constrained viewing conditions. It integrates motion-based tracking with model based face detection to produce segmented face sequences from complex scenes containing several people. The motion of moving image contours was estimated using temporal convolution and a temporally consistent list of moving objects was maintained. Objects were tracked using Kalman filters. Faces were detected using a neural network. The essence of the system is that the motion tracker is able to focus attention for a face detection network whilst the latter is used to aid the tracking process.