Despite their wide scope of application, the development of underwater models for image restoration and scene depth estimation is not a straightforward task due to the limited size and quality of underwater datasets, as well as variations in water colours resulting from attenuation, absorption and scattering phenomena in the water column. To address these challenges, we present an all-in-one conditional generative adversarial network (cGAN) called Scene-cGAN. Our cGAN is a physics-based multi-domain model designed for image dewatering, restoration and depth estimation. It comprises three generators and one discriminator. To train our Scene-cGAN, we use a multi-term loss function based on uni-directional cycle-consistency and a novel dataset. This dataset is constructed from RGB-D in-air images using spectral data and concentrations of water constituents obtained from real-world water quality surveys. This approach allows us to produce imagery consistent with the radiance and veiling light corresponding to representative water types. Additionally, we compare Scene-cGAN with current state-of-the-art methods using various datasets. Results demonstrate its competitiveness in terms of colour restoration and its effectiveness in estimating the depth information for complex underwater scenes. • We present an all-in-one conditional generative adversarial network for underwater computer vision. • Our approach addresses underwater restoration and depth estimation using three generators. • We present an underwater physics-based benchmark dataset. • Experimental results show our approach is competitive with other restoration methods and provides accurate depth estimations, aligning well with the scene.

• Novel routing method uses label hierarchies to guide capsule networks. • Capsule connections adapt based on relationships in the class hierarchy. • Agreement step enforces consistency across parent-child label levels. • Outperforms prior methods on six datasets with stronger hierarchy capture. Hierarchical multi-label classification in computer vision presents significant challenges in maintaining consistency across different levels of class granularity while capturing fine-grained visual details. This paper presents Taxonomy-aware Capsule Network (HT-CapsNet), a novel capsule network architecture that explicitly incorporates taxonomic relationships into its routing mechanism to address these challenges. Our key innovation lies in a taxonomy-aware routing algorithm that dynamically adjusts capsule connections based on known hierarchical relationships, enabling more effective learning of hierarchical features while enforcing taxonomic consistency. Extensive experiments on six benchmark datasets, including Fashion-MNIST, Marine-Tree, CIFAR-10, CIFAR-100, CUB-200-2011, and Stanford Cars, demonstrate that HT-CapsNet significantly outperforms existing methods across various hierarchical classification metrics. Notably, on CUB-200-2011, HT-CapsNet achieves absolute improvements of 10.32 % , 10.2 % , 10.3 % , and 8.55 % in hierarchical accuracy, F1-score, consistency, and exact match, respectively, compared to the best-performing baseline. On the Stanford Cars dataset, the model improves upon the best baseline by 21.69 % , 18.29 % , 37.34 % , and 19.95 % in the same metrics, demonstrating the robustness and effectiveness of our approach for complex hierarchical classification tasks.

We combine online spectral clustering and contrastive representation learning into a novel deep clustering algorithm that can be used for unsupervised image classification . We estimate a spectral embedding using minibatches. Spectral cluster assignments are used by a pairwise contrastive loss to update the model’s latent space, allowing our spectral embedding to adapt over time. We obtain competitive unsupervised classification performance purely by applying K-Means to our spectral embedding. Unlike competing methods, our approach does not require strong augmentations, class-balancing penalties, offline example mining or softmax classifiers.

Covid19-twitter: A Twitter-based Dataset for Discourse Analysis in Sentence-level Sentiment Classification

Point cloud filtering is a fundamental 3D vision task, which aims to remove noise while recovering the underlying clean surfaces. State-of-the-art methods remove noise by moving noisy points along stochastic trajectories to the clean surfaces. These methods often require regularization within the training objective and/or during post-processing, to ensure fidelity. In this paper, we introduce StraightPCF, a new deep learning based method for point cloud filtering. It works by moving noisy points along straight paths, thus reducing discretization errors while ensuring faster convergence to the clean surfaces. We model noisy patches as intermediate states between high noise patch variants and their clean counterparts, and design the VelocityModule to infer a constant flow velocity from the former to the latter. This constant flow leads to straight filtering trajectories. In addition, we introduce a DistanceModule that scales the straight trajectory using an estimated distance scalar to attain convergence near the clean surface. Our network is lightweight and only has ~530K parameters, being 17% of IterativePFn (a most recent point cloud filtering network). Extensive experiments on both synthetic and real-world data show our method achieves state-of-the-art results. Our method also demonstrates nice distributions of filtered points without the need for regularization. The implementation code can be found at: https://github.com/ddsediri/StraightPCF.

Disseminating prior knowledge about a pattern recognition task in Deep Neural Networks (DNNs) is desirable, to enable them to learn some complex patterns or representations, that are otherwise difficult to learn via usual data-driven training. Several methods have been proposed for that purpose, but creating an end-to-end trainable DNN model, while keeping it informed with prior knowledge, remains a challenging task. In this paper, we propose a method to disseminate prior knowledge in DNN models. Specifically, we created a novel MAsk-based pRior Knowledge diSsemination mechanism (MARKS-mech), that transfers logical prior knowledge in DNN models via input data transformation. We utilize a recently constructed Twitter-based dataset to perform our experiments, which is specifically designed to test the logical prior knowledge dissemination ability of methods like ours. We find that our method provides superior knowledge dissemination performance compared to the baselines.

Point Cloud Normal Estimation via Representation Learning on Height Maps

Recent advances in vision-language models have resulted in improved accuracy in visual question answering (VQA) tasks. However, their robustness remains limited when faced with out-of-distribution data containing unanswerable questions. In this study, we first construct a simple randomised VQA dataset, incorporating unanswerable questions from the VQA v2 dataset, to evaluate the robustness of a state-of-the-art VQA model. Our findings reveal that the model struggles to predict the “unknown” answer or provides inaccurate responses with high confidence scores for irrelevant questions. To address this issue without retraining the large backbone models, we propose Cross Modal Augmentation (CMA), a model-agnostic, test-time-only, multi-modal semantic augmentation technique. CMA generates multiple semantically-consistent but heterogeneous instances from the visual and textual inputs, which are then fed to the model, and the predictions are combined to achieve a more robust output. We demonstrate that implementing CMA enables the VQA model to provide more reliable answers in scenarios involving unanswerable questions, and show that the approach is generalisable across different categories of pre-trained vision language models.

A consistency-aware deep capsule network for hierarchical multi-label image classification

Abstract Squamous cell carcinoma is the most common type of cancer that occurs in squamous cells of epithelial tissue. Histopathological evaluation of tissue samples is the gold standard approach used for carcinoma diagnosis. SCC detection based on various histopathological features often employs traditional machine learning approaches or pixel-based deep CNN models. This study aims to detect keratin pearl, the most prominent SCC feature, by implementing RetinaNet one-stage object detector. Further, we enhance the model performance by incorporating an attention module. The proposed method is more efficient in detection of small keratin pearls. This is the first work detecting keratin pearl resorting to the object detection technique to the extent of our knowledge. We conducted a comprehensive assessment of the model both quantitatively and qualitatively. The experimental results demonstrate that the proposed approach enhanced the mAP by about 4% compared to default RetinaNet model.

Point cloud filtering and normal estimation are two fundamental research problems in the 3D field. Existing methods usually perform normal estimation and filtering separately and often show sensitivity to noise and/or inability to preserve sharp geometric features such as corners and edges. In this article, we propose a novel deep learning method to jointly estimate normals and filter point clouds. We first introduce a 3D patch based contrastive learning framework, with noise corruption as an augmentation, to train a feature encoder capable of generating faithful representations of point cloud patches while remaining robust to noise. These representations are consumed by a simple regression network and supervised by a novel joint loss, simultaneously estimating point normals and displacements that are used to filter the patch centers. Experimental results show that our method well supports the two tasks simultaneously and preserves sharp features and fine details. It generally outperforms state-of-the-art techniques on both tasks.

The quality of point clouds is often limited by noise introduced during their capture process. Consequently, a fundamental 3D vision task is the removal of noise, known as point cloud filtering or denoising. State-of-the-art learning based methods focus on training neural networks to infer filtered displacements and directly shift noisy points onto the underlying clean surfaces. In high noise conditions, they iterate the filtering process. However, this iterative filtering is only done at test time and is less effective at ensuring points converge quickly onto the clean surfaces. We propose IterativePFN (iterative point cloud filtering network), which consists of multiple IterationModules that model the true iterative filtering process internally, within a single network. We train our IterativePFn network using a novel loss function that utilizes an adaptive ground truth target at each iteration to capture the relationship between intermediate filtering results during training. This ensures that the filtered results converge faster to the clean surfaces. Our method is able to obtain better performance compared to state-of-the-art methods. The source code can be found at: https://github.com/ddsediri/IterativePFN

Existing normal estimation methods for point clouds are often less robust to severe noise and complex geometric structures. Also, they usually ignore the contributions of different neighbouring points during normal estimation, which leads to less accurate results. In this paper, we introduce a weighted normal estimation method for 3D point cloud data. We innovate in two key points: 1) we develop a novel weighted normal regression technique that predicts point-wise weights from local point patches and use them for robust, feature-preserving normal regression; 2) we propose to conduct contrastive learning between point patches and the corresponding ground-truth normals of the patches’ central points as a pre-training process to facilitate normal regression. Comprehensive experiments demonstrate that our method can robustly handle noisy and complex point clouds, achieving state-of-the-art performance on both synthetic and real-world datasets.

The problem of statistical heterogeneity in Federated Learning has been a major challenge, with existing solutions making unrealistic assumptions about the availability of shared datasets and high bandwidth between clients and the server. Solving this problem is crucial for the success of Federated Learning in real-world scenarios. In this work, we propose a biologically inspired dual-network model FedDual, which mimics how the human brain learns and memorizes the information. The model consists of a neocortical and a hippocampal network similar to those in the human brain. The hippocampal network is comprised by an image classification model, while the neocortical network is a variational auto-encoder responsible for long-term and re-callable memory. In this manner, FedDual uses the neocortical network to generate pseudo-patterns (synthetic data) on the server (global model). This allows for the hippocampal network to be trained with these pseudo-patterns. The dual-network architecture allows devices to share information via the weight updates of the neocortical network to the server without sending the actual data. We compare FedDual against alternatives elsewhere in the literature when applied to widely available datasets. FedDual not only achieves a margin of accuracy improvement over the alternatives, but also converges faster, requiring less communication rounds.

Recurrent Neural Networks (RNN) are widespread for Video Super-Resolution (VSR) because of their proven ability to learn spatiotemporal inter-dependencies across the temporal dimension. Despite RNN’s ability to propagate memory across longer sequences of frames, vanishing gradient and error accumulation remain major obstacles to unidirectional RNNs in VSR. Several bi-directional recurrent models are suggested in the literature to alleviate this issue; however, these models are only applicable to offline use cases due to heavy demands for computational resources and the number of frames required per input. This paper proposes a novel unidirectional recurrent model for VSR, namely “Replenished Recurrency with Dual-Duct” (R2D2), that can be used in an online application setting. R2D2 incorporates a recurrent architecture with a sliding-window-based local alignment resulting in a recurrent hybrid architecture. It also uses a dual-duct residual network for concurrent and mutual refinement of local features along with global memory for full utilisation of the information available at each timestamp. With novel modelling and sophisticated optimisation, R2D2 demonstrates competitive performance and efficiency despite the lack of information available at each time-stamp compared to its offline (bi-directional) counterparts. Ablation analysis confirms the additive benefits of the proposed sub-components of R2D2 over baseline RNN models.The PyTorch-based code for the R2D2 model will be released at R2D2 GitRepo.

Disseminating and incorporating logic rules into deep neural networks has been extensively explored for sentiment classification in recent years. In particular, most methods and algorithms proposed for this purpose rely on a specific component that aims to capture and model logic rules, followed by a sequence model to process the input sequence. While the authors of these methods claim that they effectively capture syntactic structures that affect sentiment classification, they only show improvement in accuracy to support their claims without further analysis. Focusing on various syntactic structures, particularly contrastive discourse relations such as the A-but-B structure, we introduce the PERCY score, a novel Post-hoc Explanation-based Rule ConsistencY Score to analyze and study the ability of several of these methods to identify these structures in a given sentence, and to make their classification decisions based on the appropriate conjunct. Specifically, we explore the use of model-agnostic post-hoc explanation frameworks to explain the predictions of any classifier in an interpretable and faithful manner. These model explainability frameworks provide feature attribution scores to estimate each word’s impact on the final classification decision. Then, they are combined to check whether the model has based its decision on the right conjunct. Our experiments show that (a) accuracy – or any other performance metric – can be misleading in assessing the ability of logic rule dissemination methods to base their decisions on the right conjunct, (b) not all analyzed methods effectively capture syntactic structures, (c) often, the underlying sequence model is what captures the structure, and (d) for the best method, less than 25% of the test examples are classified based on the appropriate conjunct, indicating that a lot of research needs to be done on this topic. Finally, we experimentally demonstrate that the PERCY scores calculated are robust and stable w.r.t. the feature-attribution frameworks used.

Spectral graph convolutional neural networks (GCNNs) have been producing encouraging results in graph classification tasks. However, most spectral GCNNs utilize fixed graphs when aggregating node features, while omitting edge feature learning and failing to get an optimal graph structure. Moreover, many existing graph datasets do not provide initialized edge features, further restraining the ability of learning edge features via spectral GCNNs. In this paper, we try to address this issue by designing an edge feature scheme and an add-on layer between every two stacked graph convolution layers in spectral GCNN. Both are lightweight while effective in filling the gap between edge feature learning and performance enhancement of graph classification. The edge feature scheme makes edge features adapt to node representations at different spectral graph convolution layers. The add-on layers help adjust the edge features to an optimal graph structure. To test the effectiveness of our method, we take Euclidean positions as initial node features and extract graphs with semantic information from point cloud objects. The node features of our extracted graphs are more scalable for edge feature learning than most existing graph datasets (in one-hot encoded label format). Three new graph datasets are constructed based on ModelNet40, ModelNet10 and ShapeNet Part datasets. Experimental results show that our method outperforms state-of-the-art graph classification methods on the new datasets. Our code and the constructed graph datasets will be released to the community.

This paper presents Marine-tree, a large-scale hierarchical annotated dataset for marine organism classification. Marine-tree contains more than 160k annotated images divided into 60 classes organised in a hierarchy-tree structure using an adapted CATAMI (Collaborative and Automated Tools for the Analysis of Marine Imagery and video) classification scheme. Images were meticulously collected by scuba divers using the RLS (Reef Life Survey) methodology and later annotated by experts in the field. We also propose a hierarchical loss function that can be applied to any multi-level hierarchical classification model, which takes into account the parent-child relationship between predictions and uses it to penalize inconsistent predictions. Experimental results demonstrate thatMarine-tree and the proposed hierarchical loss function are a good contribution for both research in underwater imagery and hierarchical classification.

Current normal estimation methods for 3D point clouds often show limited accuracy in predicting normals at sharp features (e.g., edges and corners) and less robustness to noise. In this paper, we propose a novel normal estimation method for point clouds which consists of two phases: (a) feature encoding to learn representations of local patches, and (b) normal estimation that takes the learned representation as input and regresses the normal vector. We are motivated that local patches on isotropic and anisotropic surfaces respectively have similar and distinct normals, and these separable features or representations can be learned to facilitate normal estimation. To realise this, we design a triplet learning network for feature encoding and a normal estimation network to regress normals. Despite having a smaller network size compared with most other methods, experiments show that our method preserves sharp features and achieves better normal estimation results especially on computer-aided design (CAD) shapes.

In this paper, we present T-Net, a fully convolutional network particularly well suited for resource constrained and mobile devices, which cannot cater for the computational resources often required by much larger networks. T-NET’s design allows for dual-stream information flow both inside as well as outside of the encoder-decoder pair. Here, we use group convolutions to increase the width of the network and, in doing so, learn a larger number of low and intermediate level features. We have also employed skip connections in order to keep spatial information loss to a minimum. T-Net uses a dice loss for pixel-wise classification which alleviates the effect of class imbalance. We have performed experiments with three different applications, retinal vessel segmentation, skin lesion segmentation and digestive tract polyp segmentation. In our experiments, T-Net is quite competitive, outperforming alternatives with two or even three orders of magnitude more trainable parameters.

Deep neural networks tend to underestimate uncertainty and produce overly confident predictions. Recently proposed solutions, such as MC Dropout and SDENet, require complex training and/or auxiliary out-of-distribution data. We propose a simple solution by extending the time-tested iterative reweighted least square (IRLS) in generalised linear regression. We use two sub-networks to parametrise the prediction and uncertainty estimation, enabling easy handling of complex inputs and nonlinear response. The two sub-networks have shared representations and are trained via two complementary loss functions for the prediction and the uncertainty estimates, with interleaving steps as in a cooperative game. Compared with more complex models such as MC-Dropout or SDE-Net, our proposed network is simpler to implement and more robust (insensitive to varying aleatoric and epistemic uncertainty).

Rip currents are intense and localized seaward-flowing water from shore through the surf zone, cutting through the lines of breaking ocean waves. It generally pulls a person out to sea very fast, with speeds up to eight feet per second. It is one of the most important reasons for an average of 21 confirmed fatalities each year. Fully automated Rip current detection using state-of-the-art deep learning techniques can help to monitor the coast. However, there are several challenges involved, like dealing with a small training sample size and unavailability of pretrained models in the domain of Rip current. In this work, we address these issues and propose a novel fast and lightweight RipDet framework for an efficient and accurate automated Rip current detection. The proposed model is aided by careful data augmentation, fine-tuning, and a custom learning rate schedule that helps the model adapt to Rip currents' distribution with the low number of training samples. Extensive experiments on benchmark data show that the proposed model outperforms other state-of-the-art methods for Rip current detection with mean average precision (mAP) score of 98.131 %.

Light field cameras (LFCs) have received increasing attention due to their wide-spread applications. However, current LFCs suffer from the well-known spatio-angular trade-off, which is considered an inherent and fundamental limit for LFC designs. In this article, by doing a detailed optical analysis of the sampling process in an LFC, we show that the effective resolution is generally higher than the number of micro-lenses. This contribution makes it theoretically possible to super-resolve a light field. Further optical analysis proves the "2D predictable series" nature of the 4D light field, which provides new insights for analyzing light field using series processing techniques. To model this nature, a specifically designed epipolar plane image (EPI) based CNN-LSTM network is proposed to super-resolve a light field in the spatial and angular dimensions simultaneously. Rather than leveraging semantic information, our network focuses on extracting geometric continuity in the EPI domain. This gives our method an improved generalization ability and makes it applicable to a wide range of previously unseen scenes. Experiments on both synthetic and real light fields demonstrate the improvements over state-of-the-arts, especially in large disparity areas.

Deep learning models have attracted considerable attention in metamodelling of financial risks for large insurance portfolios. Those models, however, are generally trained in disregard of the collective nature of the data in the portfolio under study. Consequently, the training procedure often suffers from slow convergence, and the trained model often has poor accuracy. This is particularly evident in the presence of extreme individual contracts. In this paper, we advocate the view that the training of a meta-model for a portfolio should be guided by portfolio-level metrics. In particular, we propose an intuitive loss regulariser that explicitly accounts for the portfolio-level bias. Further, this training regulariser can be easily implemented with the minibatch stochastic gradient descent commonly used in training deep neural networks. Empirical evaluations on both simulated data and a benchmark dataset show that the regulariser yields more stable training, resulting in faster convergence and more reliable portfolio-level risk estimates.

This paper introduces a novel method to simultaneously super-resolve and colour-predict images acquired by snapshot mosaic sensors. These sensors allow for spectral images to be acquired using low-power, small form factor, solid-state CMOS sensors that can operate at video frame rates without the need for complex optical setups. Despite their desirable traits, their main drawback stems from the fact that the spatial resolution of the imagery acquired by these sensors is low. Moreover, chromatic mapping in snapshot mosaic sensors is not straightforward since the bands delivered by the sensor tend to be narrow and unevenly distributed across the range in which they operate. We tackle this drawback as applied to chromatic mapping by using a residual channel attention network equipped with a texture sensitive block. Our method significantly outperforms the traditional approach of interpolating the image and, afterwards, applying a colour matching function. This work establishes state-of-the-art in this domain while also making available to the research community a dataset containing 296 registered stereo multi-spectral/RGB images pairs.

Photometric stereo estimates the surface normal given a set of images acquired under different illumination conditions. To deal with diverse factors involved in the image formation process, recent photometric stereo methods demand a large number of images as input. We propose a method that can dramatically decrease the demands on the number of images by learning the most informative ones under different illumination conditions. To this end, we use a deep learning framework to automatically learn the critical illumination conditions required at input. Furthermore, we present an occlusion layer that can synthesize cast shadows, which effectively improves the estimation accuracy. We assess our method on challenging real-world conditions, where we outperform techniques elsewhere in the literature with a significantly reduced number of light conditions.

Internet of Things (IoT) has an immense potential for a plethora of applications ranging from healthcare automation to defence networks and the power grid. The security of an IoT network is essentially paramount to the security of the underlying computing and communication infrastructure. However, due to constrained resources and limited computational capabilities, IoT networks are prone to various attacks. Thus, safeguarding the IoT network from adversarial attacks is of vital importance and can be realised through planning and deployment of effective security controls; one such control being an intrusion detection system. In this paper, we present a novel intrusion detection scheme for IoT networks that classifies traffic flow through the application of deep learning concepts. We adopt a newly published IoT dataset and generate generic features from the field information in packet level. We develop a feed-forward neural networks model for binary and multi-class classification including denial of service, distributed denial of service, reconnaissance and information theft attacks against IoT devices. Results obtained through the evaluation of the proposed scheme via the processed dataset illustrate a high classification accuracy.

Here, we present a pixelwise illuminant recovery method for both, trichromatic and multi or hyperspectral images which employs a convolutional neural nettwork. The network used here is based upon the simple, yet effective architecture employed by the CIFARIO-quick net[1]. The network is trained using a loss function which employs the angular difference between the target illuminant and the estimated one as the data term. The loss used here also includes a regularisation term which encourages smoothness in the spectral domain. Moreover, the network takes, at input, a tensor which is constructed making use of an image patch at different scales. This allows the network to predict the illuminant per-pixel using locally supported multiscale information. We illustrate the utility of our method for both, colour and hyperspectal illuminant recovery and compare our results against other techniques elsewhere in literature.

In this paper, we present a frequency domain neural network for image super-resolution. The network employs the convolution theorem so as to cast convolutions in the spatial domain as products in the frequency domain. Moreover, the non-linearity in deep nets, of ten achieved by a rectifier unit, is here cast as a convolution in the frequency domain. This not only yields a network which is very computationally efficient at testing, but also one whose parameters can all be learnt accordingly. The network can be trained using back propagation and is devoid of complex numbers due to the use of the Hartley transform as an alternative to the Fourier transform. Moreover, the network is potentially applicable to other problems elsewhere in computer vision and image processing which are of ten cast in the frequency domain. We show results on super-resolution and compare against alternatives elsewhere in the literature. In our experiments, our network is one to two orders of magnitude faster than the alternatives with a marginal loss of performance.

In this paper, we present a spectral clustering method for online and streaming applications. Here, we note that the rank of the coefficients of the eigenvector of the graph Laplacian govern, together with the weights of the adjacency matrix, the assignment of the data to clusters. Thus, we adopt a sampling without replacement strategy, where, at each sampling step, we select those data instances which are most relevant to the clustering process. To do this, we “sparsify” the eigenvector making use of a Minorisation-Maximisation approach. This not only allows to cluster the data under consideration after the sampling has been effected, but also permits the optimisation in hand to be performed making use of a gradient descent approach with a closed form iterate. Moreover, the method presented here is quite general in nature and can be employed in other settings which hinge in an L-0 regularised penalty function. We discuss the use of our approach for the assessment of node centrality and document binarisation. We also illustrate the utility of our method for purposes of background subtraction and compare our results with those yielded by alternatives elsewhere in the literature.

In this paper we present a method to recover the shading and specularities in the scene from a single image. The method presented here is based on the dichromatic model and enforces a local smoothness assumption over the object surfaces in the scene. This naturally leads to a setting where the estimate of the shading at a particular pixel can be expressed in terms of its neighbours up to a pair of Gaussian kernels accounting for the irradiance similarity between pixels and their spatial proximity on the image plane. This yields a quadratic cost function for both, the specular coefficient and the shading factor of the dicromatic model which can be solved using gradient descent. We show results for both, specular highlight recovery and shading estimation and compare them against a number of alternatives.

Here, we turn our attention to barycentric embeddings and examine their utility for semi-supervised image labelling tasks. To this end, we view the pixels in the image as vertices in a graph and their pairwise affinities as weights of the edges between them. Abstracted in this manner, we can pose the semi-supervised labelling problem into a graph theoretic setting where the labels are assigned based upon the distance in the embedding space between the nodes corresponding to the unlabelled pixels and those whose labels are in hand. We do this using a barycentric embedding approach which naturally leads to a setting in which the embedding coordinates can be computed by solving a system of linear equations. Moreover, the method presented here can incorporate side information such as that delivered by colour priors used elsewhere in the literature for semi-supervised colour image labelling. We illustrate the utility of our method for colour image labelling and material classification on hyperspectal images. We also compare our results against other techniques elsewhere in literature.

In this paper, we employ graph embeddings for classification tasks. To do this, we explore the relationship between kernel matrices, spaces of inner products and statistical inference by viewing the embedding vectors for the nodes in the graph as a field on a Riemannian manifold. This leads to a setting where the inference process may be cast as a Maximum a Posteriori (MAP) estimation over a Gibbs field whereby the graph Laplacian can be related to a Gram matrix of scalar products. This not only allows for a better understanding of graph spectral techniques, but also provides a means for classifying nodes in the graph without the need to compute the embedding explicitly by using a Mercer kernel. We illustrate how the developments presented here can be used for purposes of classification, where we use the graph Laplacian as a kernel matrix. We present classification results on synthetic data and four UCI datasets. We also apply our method to real-world image labelling and compare our results to those yielded by alternatives elsewhere in the literature.

This paper presents a method to recover a spatially varying illuminant color estimate from scenes lit by multiple light sources. Starting with the image formation process, we formulate the illuminant recovery problem in a statistically data-driven setting. To do this, we use a factor graph defined across the scale space of the input image. In the graph, we utilize a set of illuminant prototypes computed using a data driven approach. As a result, our method delivers a pixelwise illuminant color estimate being devoid of libraries or user input. The use of a factor graph also allows for the illuminant estimates to be recovered making use of a maximum a posteriori inference process. Moreover, we compute the probability marginals by performing a Delaunay triangulation on our factor graph. We illustrate the utility of our method for pixelwise illuminant color recovery on widely available data sets and compare against a number of alternatives. We also show sample color correction results on real-world images.

In this paper we present a method to estimate the radiance map and camera response function for high dynamic range (HDR) imaging which is devoid of free parameters and can reliably recover HDR images with as few as two low-dynamic range (LDR) views acquired at different exposures. To do this, we employ a L1 cost function which lends itself to the use of a Weiszfeld optimisation scheme. We illustrate the utility of the method for computing high-dynamic range images from low-dynamic range imagery. In our experiments, we compare our results with those delivered by alternatives elsewhere in the literature and show that our method can outperform the alternatives with as few as two LDR images. We also apply our method to panorama generation.

This paper presents a method to recover the pixel-wise illuminant colour for scenes lit by multiple lights. Here, we start from the image formation process and pose the illuminant recovery task in hand into an evidence combining setting. To do this, we construct a factor graph making use of the scale space of the input image and a set of illuminant prototypes. The computation of these prototypes is data driven and, hence, our method is devoid of libraries or user input. The use of a factor graph allows for the illuminant estimates at different scales to be recovered making use of a maximum a posteriori (MAP) inference process. Moreover, we render the computation of the probability marginals used here as exact by constructing our factor graph making use of a Delaunay triangulation. We illustrate the utility of our method for pixelwise illuminant colour recovery on two widely available datasets and compare against a number of alternatives. We also show sample colour correction results on real-world images.

In this paper, we present a method which can perform spectral reconstruction and illuminant recovery from a single colour image making use of an unlabelled training set of hyperspectral images. Our method employs colour and appearance information to drive the reconstruction process subject to the material properties of the objects in the scene. The idea is to reconstruct the image spectral irradiance making use of a set of prototypes extracted from the training set. These spectra, together with a set of convolutional features are hence obtained using sparse coding so as to reconstruct the image irradiance. With the reconstructed spectra in hand, we proceed to compute the illuminant power spectrum using a quadratic optimisation approach. We provide a quantitative analysis for our method and compare to a number of alternatives. We also show sample results on illuminant substitution and transfer, film simulation and image recolouring using mood board colour schemes.

This paper proposes a method to approximate the camera spectral sensitivity functions from a single colour image of a colour chart under an unknown illumination spectrum. Here we assume that the scene illumination has a smooth spectral variation. Although the original problem is rather ill-posed, we reformulate it as a well-posed optimisation one by introducing several constraints. The first constraint concerns with the smoothness of the illuminant power spectrum. The second one is based on the fact that the spectral sensitivity function of a digital camera should be constrained to a linear subspace according to Luther condition. The third constraint limits the influence of bands with low signal-to-noise ratios. By introducing these constraints, we can solve the problem in a coordinate-descent optimisation manner. We validate our method using data acquired from over 40 commercial camera models and compare with an alternative in the literature. We also demonstrate the utility of the estimated colour matching functions for colour simulation and colour transfer.

Here, we turn our attention to factor graphs and examine their message passing properties for image processing tasks. To this end, we focus on the maximum a posteriori (MAP) inference process in multi-layered graphs and exploit the ability of factor graphs to capture subtle interactions between image tokens, i.e. pixels, super pixels, features, etc. This leads to a general, yet simple belief propagation scheme. The benefits of doing this are two-fold. Firstly, this yields the ability to perform more accurate joint probability inference tasks at minimal additional computational cost. Secondly, we gain the advantage of modelling structural interactions between image tokens more accurately on graphical models with multiple levels of interaction (layers). We illustrate the use of factor graphs for image defogging and segmentation and compare our results against other techniques elsewhere in literature.

In this paper we propose a method for learning the materials in a scene in an unsupervised manner making use of imaging spectroscopy data. Here, we view the input image spectra as a data point on a manifold which corresponds to a node in a graph whose vertices correspond to a set of parameters that should be inferred using the Expectation Maximisation (EM) algorithm. In this manner, we can pose the problem as a statistical unsupervised learning one where the aim of computation becomes the recovery of the set of parameters that allow for the image spectra to be projected onto a set of graph vertices defined a priori. Moreover, as a result of this treatment, the scene material prototypes can be recovered making use of a clustering algorithm applied to the parameter-set. This setting also allows, in a straightforward manner, for the visualisation of the spectra. We discuss the links between our method and self-organizing maps and illustrate the utility of the method as compared to other alternatives elsewhere in the literature.

In this paper, we present an unsupervised method for segmenting the illuminant regions and estimating the illumination power spectrum from a single image of a scene lit by multiple light sources. Here, illuminant region segmentation is cast as a probabilistic clustering problem in the image spectral radiance space. We formulate the problem in an optimization setting, which aims to maximize the likelihood of the image radiance with respect to a mixture model while enforcing a spatial smoothness constraint on the illuminant spectrum. We initialize the sample pixel set under each illuminant via a projection of the image radiance spectra onto a low-dimensional subspace spanned by a randomly chosen subset of spectra. Subsequently, we optimize the objective function in a coordinate-ascent manner by updating the weights of the mixture components, sample pixel set under each illuminant, and illuminant posterior probabilities. We then estimate the illuminant power spectrum per pixel making use of these posterior probabilities. We compare our method with a number of alternatives for the tasks of illumination region segmentation, illumination color estimation, and color correction. Our experiments show the effectiveness of our method as applied to one hyperspectral and three trichromatic image data sets.

In this paper, we present a method to recover the albedo and depth from a single image. To this end, we depart from the scattering theory in the atmospheric vision model used elsewhere for defogging and dehazing. We then view the image as a relaxed factorial Markov random field (FMRF) of albedo and depth layers. This leads to a formulation which, for each of the layers in the FMRF, is akin to relaxation labelling problems. Moreover, we can obtain sparse representations for the graph Laplacian and Hessian matrices involved. This implies that global minima for each of the layers can be estimated efficiently via sparse Cholesky factorisation methods. We illustrate the utility of our method for depth and albedo recovery making use of real world data and compare against other techniques elsewhere in the literature.

This paper presents a novel approach for estimating light direction, shape, and reflectance parameters from a single image based on iterative optimisation. We depart from a generalist view of the reflection process based upon a physical interpretation and cast the recovery of the reflection parameters in an optimisation setting. With the estimated specular reflectance parameters, we recover the light source direction from specular highlights while applying two novel constraints, coplanarity and Kullback-Leibler divergence. Then, by integrating the knowledge of light source direction and diffuse reflectance parameters, we recover shape of the scene from diffuse component. Our approach is quite general in nature and can be applied to a family of reflectance models that are based on the Fresnel reflection theory. We demonstrate the utility of our method on synthetic and real world imagery.

In this paper we present a method to automatically control the exposure of multispectral cameras. To this end, we use an input image taken using camera pre-sets. This pre-set exposure time and the input image are then used to recover an updated exposure time making use of the spectral power. This yields a setting where the spectral power is computed using a photopic function. Moreover, the exposure time can then be recovered using a bounded least squares optimisation effected upon the histogram equalised spectral power. We illustrate the utility of the method for exposure time recovery on multispectral images using a calibration chart, where we compare the timings yielded by our method against those recovered through manual photometric calibration. We also show results on real-world imagery under several lighting conditions.

In this paper, we propose a biologically inspired spiking neural network approach to obtaining an opponent pair which is invariant to illumination variations and can be employed for colour discrimination. The model is motivated by the neural mechanisms involved in processing the visual stimulus starting from the cone photo receptors to the centre-surround receptive fields present in the retinal ganglion cells and the striate cortex. For our spiking neural network, we have employed the excitatory and inhibitory lateral synaptic connections, the Spike-Timing Dependent Plasticity (STDP) and long term potentiation and depression (LTP/LTD). Here, we employ a feed-forward leaky integrate-and-fire spiking neural network trained using a dataset of Munsell spectra. We have performed tests on perceptually similar colours under large illuminant power variations and done experiments on colour-based object recognition. We have also compared our results to those yielded by a number of alternatives.

In this study, the authors explore the opportunities, application areas and challenges involving the use of imaging spectroscopy as a means for scene understanding. This is important, since scene analysis in the scope of imaging spectroscopy involves the ability to robustly encode material properties, object composition and concentrations of primordial components in the scene. The combination of spatial and compositional information opens‐up a vast number of application possibilities. For instance, spectroscopic scene analysis can enable advanced capabilities for surveillance by permitting objects to be tracked based on material properties. In computational photography, images may be enhanced taking into account each specific material type in the scene. For food security, health and precision agriculture it can be the basis for the development of diagnostic and surveying tools which can detect pests before symptoms are apparent to the naked eye. This combination of a broad domain of application with the use of key technologies makes the use of imaging spectroscopy a worthwhile opportunity for researchers in the areas of computer vision and pattern recognition.

In this paper, we address the problem of efficiently recovering reflectance parameters from a single multispectral or hyperspectral image. To do so, we propose a shapelet based estimator that employs shapelets to recover the shading in the image. The optimization setting presented is based upon a three-step process. The first of these concerns the recovery of the surface reflectance and the specular coefficients through a constrained optimization approach. Second, we update the illuminant power spectrum using a simple least-squares formulation. Third, the shading is computed directly once the updated illuminant power spectrum is obtained. This yields a computationally efficient method that achieves speed-ups of nearly an order of magnitude over its closest alternative without compromising performance. We provide results on illuminant power spectrum computation, shading recovery, skin recognition and replacement of the scene illuminant, and object reflectance in real-world images.

In this paper, we present a method which permits the creation of user colour preferences for object materials and lights in the scene making use of imaging spectroscopy data. To do this, we build upon the heterogeneous nature of the scene by imposing consistency over object materials so as to allow for small compositional variations across objects in the image. Once the consistency has been imposed, we aim at maximising the quality of the images under consideration based upon user input. This provides the flexibility necessary to utilise user profiles for the automatic processing of real world imagery while avoiding undesirable effects encountered when colour images are produced. We provide results on real-world imagery and illustrate how the method can be used to produce material-specific colours based upon user input.

Multiple instance learning (MIL) is a paradigm in supervised learning that deals with the classification of collections of instances called bags. Each bag contains a number of instances from which features are extracted. The complexity of MIL is largely dependent on the number of instances in the training data set. Since we are usually confronted with a large instance space even for moderately sized real-world data sets applications, it is important to design efficient instance selection techniques to speed up the training process without compromising the performance. In this paper, we address the issue of instance selection in MIL. We propose MILIS, a novel MIL algorithm based on adaptive instance selection. We do this in an alternating optimization framework by intertwining the steps of instance selection and classifier learning in an iterative manner which is guaranteed to converge. Initial instance selection is achieved by a simple yet effective kernel density estimator on the negative instances. Experimental results demonstrate the utility and efficiency of the proposed approach as compared to the state of the art.

In this paper, we develop a novel approach to object-material identification in spectral imaging by combining the use of invariant spectral absorption features and statistical machine-learning techniques. Our method hinges on the relevance of spectral absorption features for material identification and casts the problem into a pattern-recognition setting by making use of an invariant representation of the most discriminant band segments in the spectra. Thus, here, we view the identification problem as a classification task, which is effected based upon those invariant absorption segments in the spectra which are most discriminative between the materials under study. To robustly recover those bands that are most relevant to the identification process, we make use of discriminant learning. To illustrate the utility of our method for purposes of material identification, we perform experiments on both terrestrial and remotely sensed hyperspectral imaging data and compare our results to those yielded by an alternative.

Hyperspectral unmixing is a crucial preprocessing step for material classification and recognition. In the last decade, nonnegative matrix factorization (NMF) and its extensions have been intensively studied to unmix hyperspectral imagery and recover the material end-members. As an important constraint for NMF, sparsity has been modeled making use of the$L_{1}$regularizer. Unfortunately, the$L_{1}$regularizer cannot enforce further sparsity when the full additivity constraint of material abundances is used, hence limiting the practical efficacy of NMF methods in hyperspectral unmixing. In this paper, we extend the NMF method by incorporating the$L_{1/2}$sparsity constraint, which we name$L_{1/2}$-NMF. The$L_{1/2}$regularizer not only induces sparsity but is also a better choice among$L_{q}(0 < q < 1)$regularizers. We propose an iterative estimation algorithm for$L_{1/2}$-NMF, which provides sparser and more accurate results than those delivered using the$L_{1}$norm. We illustrate the utility of our method on synthetic and real hyperspectral data and compare our results to those yielded by other state-of-the-art methods.

In this paper, we propose an approach to the problem of simultaneous shape and refractive index recovery from multispectral polarisation imagery captured from a single viewpoint. The focus of this paper is on dielectric surfaces which diffusely polarise light transmitted from the dielectric body into the air. The diffuse polarisation of the reflection process is modelled using a Transmitted Radiance Sinusoid curve and the Fresnel transmission theory. We provide a method of estimating the azimuth angle of surface normals from the spectral variation of the phase of polarisation. Moreover, to render the problem of simultaneous estimation of surface orientation and index of refraction well-posed, we enforce a generative model on the material dispersion equations for the index of refraction. This generative model, together with the Fresnel transmission ratio, permit the recovery of the index of refraction and the zenith angle simultaneously. We show results on shape recovery and rendering for real world and synthetic imagery.

In this paper we address the problem of image fusion between imagery acquired by trichromatic sensors and hyperspectral imagers. We do this by presenting a method aimed at registering a high-resolution trichromatic image with lower resolution hyperspectral data. The method presented here maps the hyperspectral image into the grayscale image so as to employ the cross cumulative residual entropy for purposes of multimodal registration. We illustrate the utility of our approach by presenting registration results on a set of surveillance image pairs consisting of a set of high-oblique colour and hyperspectral images.

In this paper, we present a system for automatic spectral signature acquisition and recognition of skin from hyperspectral face imagery. In the acquisition step, hyperspectral cameras are used to capture multispectral or hyperspectral images of faces for skin recognition. The acquired signature may either be stored in a database for future testing or be used for purposes of identification. In the recognition step, the system accounts for variations in the illumination by recovering the light power spectrum in the scene and obtains the scene reflectance by normalising the input image radiance accordingly. Furthermore, incorporated into this system is a Non-Uniform Rational B-Spline (NURBS) compact descriptor of spectral reflectance for recognition purposes. We have employed this system as a profiling tool to classify a real-world multispectral face image database into separate ethnic groups.

In this paper, we present an approach to robust estimation of shape from single-view multi-spectral polarisation images. The developed technique tackles the problem of recovering the azimuth angle of surface normals robust to image noise and a low degree of polarisation. We note that the linear least-squares estimation results in a considerable phase shift from the ground truth in the presence of noise and weak polarisation in multispectral and hyper spectral imaging. This paper discusses the utility of robust statistics to discount the large error attributed to outliers and noise. Combining this approach with Shape from Shading, we fully recover the surface shape. We demonstrate the effectiveness of the robust estimator compared to the linear least-squares estimator through shape recovery experiments on both synthetic and real images.

In this paper, we tackle the problem of image in painting which aims at removing objects from an image or repairing damaged pictures by replacing the missing regions using the information in the rest of the scene. The image in painting method proposed here builds on an exemplar-based perspective so as to improve the local consistency of the in painted region. This is done by selecting the optimal patch which maximises the local consistency with respect to abutting candidate patches. The similarity computation generates weights based upon an edge prior and the structural differences between in painting exemplar candidates. This treatment permits the generation of an in painting sequence based on a list of factors. The experiments show that the proposed method delivers a margin of improvement as compared to alternative methods.

Multi-sensor image registration is a challenging task in remote sensing. Considering the fact that multi-sensor devices capture the images at different times, multi-spectral image registration is necessary for data fusion of the images. Several conventional methods for image registration suffer from poor performance due to their sensitivity to scale and intensity variation. The scale invariant feature transform (SIFT) is widely used for image registration and object recognition to address these problems. However, directly applying SIFT to remote sensing image registration often results in a very large number of feature points or keypoints but a small number of matching points with a high false alarm rate. We argue that this is due to the fact that spatial information is not considered during the SIFT-based matching process. This paper proposes a method to improve SIFT-based matching by taking advantage of neighborhood information. The proposed method generates more correct matching points as the relative structure in different remote sensing images are almost static.

In this paper, we address the problem of combining linear support vector machines (SVMs) for classification of large-scale nonlinear datasets. The motivation is to exploit both the efficiency of linear SVMs (LSVMs) in learning and prediction and the power of nonlinear SVMs in classification. To this end, we develop a LSVM mixture model that exploits a divide-and-conquer strategy by partitioning the feature space into subregions of linearly separable datapoints and learning a LSVM for each of these regions. We do this implicitly by deriving a generative model over the joint data and label distributions. Consequently, we can impose priors on the mixing coefficients and do implicit model selection in a top-down manner during the parameter estimation process. This guarantees the sparsity of the learned model. Experimental results show that the proposed method can achieve the efficiency of LSVMs in the prediction phase while still providing a classification performance comparable to nonlinear SVMs.

Multiple-instance learning (MIL) is a new paradigm of supervised learning that deals with the classification of bags. Each bag is presented as a collection of instances from which features are extracted. In MIL, we have usually confronted with a large instance space for even moderately sized data sets since each bag may contain many instances. Hence it is important to design efficient instance pruning and selection techniques to speed up the learning process without compromising on the performance. In this paper, we address the issue of instance selection in multiple instance learning and propose the IS-MIL, an instance selection framework for MIL, to tackle large-scale MIL problems. IS-MIL is based on an alternative optimisation framework by iteratively repeating the steps of instance selection/updating and classifier learning, which is guaranteed to converge. Experimental results demonstrate the utility and efficiency of the proposed approach compared to the alternatives.

In this paper, we identify the constraints under which the generally ill-posed problem of simultaneous recovery of surface shape and its photometric invariants can be rendered tractable. We examine the cases where a single or more images are acquired using different lighting directions with known illuminant power. Given these conditions, we state the constraints upon which the recovery of the surface geometry and its photometric parameters can be estimated. With these constraints, we then show how the recovery process may be formulated as an optimisation algorithm which aims to fit the reflectance models under study to the image reflectance. The approach presented here is general and can be applied to a family of reflectance models that are based on the Fresnel reflection theory. Thus, we provide a theoretical and computational background for recovering shape, material index of refraction and microscopic roughness from multi-spectral images.

In this paper, we present a surface reflectance descriptor based on the control points resulting from the interpolation of Non-Uniform Rational B-Spline (NURBS) curves to multispectral reflectance data. The interpolation is based upon a knot removal scheme in the parameter domain. Thus, we exploit the local support of NURBS so as to recover a compact descriptor robust to noise and local perturbation of the spectra. We demonstrate the utility of our NURBS-based descriptor for material identification. To this end, we perform skin spectra recognition making use of a Support Vector Machine classifier. We also provide results on hyperspectral imagery and elaborate on the preprocessing step for skin segmentation. We compare our results with those obtained using an alternative descriptor.

In this paper, we present a novel approach to contents-based image retrieval. The method hinges in the use of quasi-random sampling to retrieve those images in a database which are related to a query image provided by the user. Departing from random sampling theory, we make use of the EM algorithm so as to organize the images in the database into compact clusters that can then be used for stratified random sampling. For the purposes of retrieval, we use the similarity between the query and the clustered images to govern the sampling process within clusters. In this way, the sampling can be viewed as a stratified sampling one which is random at the cluster level and takes into account the intra-cluster structure of the dataset. This approach leads to a measure of statistical confidence that relates to the theoretical hard-limit of the retrieval performance. We show results on the Oxford Flowers dataset.

In this paper, we develop an efficient logistic regression model for multiple instance learning that combines L1and L2regularisation techniques. An L1regularised logistic regression model is first learned to find out the sparse pattern of the features. To train the L1model efficiently, we employ a convex differentiable approximation of the L1cost function which can be solved by a quasi Newton method. We then train an L2regularised logistic regression model only on the subset of features with nonzero weights returned by the L1logistic regression. Experimental results demonstrate the utility and efficiency of the proposed approach compared to a number of alternatives.

In this paper, we propose a hierarchical approach to image segmentation based on the use of a graph regularisation algorithm. The initial segmentation map is obtained using the normalized cut segmentation algorithm. We then refine the segmentation results by iteratively propagating the class-labels from coarse-to-fine sampling levels. Image segmentation at each intermediate level is recast as a constrained graph regularisation problem that can be solved efficiently. The multi-level nature of our method achieves low computational cost and robustness to noise corruption. We provide experimental results on the Berkeley Image Database and show the efficacy of our method for segmentation of high resolution images.

In this paper, we present a semi supervised approach to space carving. We do this by casting the recovery of volumetric data from multiple views into an evidence combining setting. The method presented here is statistical in nature and employs, as a starting point, a manually obtained contour. By making use of this user-provided information, we learn a prior distribution that is then used to compute the probability of a voxel being carved. This evidence combining setting allows us to make use of background pixel information. As a result, our method combines the advantages of shape-from-silhouette techniques and statistical space carving approaches. We provide quantitative results and illustrate the utility of the method on real-world imagery.

In this paper, we investigate linear discriminant analysis (LDA) methods for multiclass classification problems in hyperspectral imaging. We note that LDA does not consider pairwise relations between different classes, it rather assumes equal within and between-class scatter matrices. As a result, we present a pairwise discriminant analysis algorithm for learning class categories. Our pairwise linear discriminant analysis measures the separability of two classes making use of the class centroids and variances. Our approach is based upon a novel cost function with unitary constraints based on the aggregation of pairwise costs for binary classes. We view the minimisation of this cost function as an unconstrained optimisation problem over a Grassmann manifold and solve using a projected gradient method. Our approach does not require matrix inversion operations and, therefore, does not suffer of stability problems for small training sets. We demonstrate the utility of our algorithm for purposes of learning material catergories in hyperspectral images.

In this paper, we present a probabilistic formulation of kernel-based tracking methods based upon maximum likelihood estimation. To this end, we view the coordinates for the pixels in both, the target model and its candidate as random variables and make use of a generative model so as to cast the tracking task into a maximum likelihood framework. This, in turn, permits the use of the EM-algorithm to estimate a set of latent variables that can be used to update the target-center position. Once the latent variables have been estimated, we use the Kullback-Leibler divergence so as to minimise the mutual information between the target model and candidate distributions in order to develop a target-center update rule and a kernel bandwidth adjustment scheme. The method is very general in nature. We illustrate the utility of our approach for purposes of tracking on real-world video sequences using two alternative kernel functions.

In this paper, we aim at presenting a survey on automatic absorption recovery methods for imaging spectroscopy. We commence by viewing the algorithms in the literature from a technical perspective and presenting an overview of the derivative analysis, fingerprint, and maximum modulus wavelet transform techniques. In addition to these methods, we also present a novel absorption recovery approach based upon unimodal regression and continuum removal. With this technical review of the methods under study, we perform a complexity analysis and examine the implementation issues pertaining to each of the alternatives. We show how detected absorption bands can be used for purposes of material identification. We conclude this paper by providing a performance study and providing identification results on hyperspectral imagery. To this end, we make use of a number of distance measures to evaluate the quality of the recovered absorptions, as compared to continuum-removed spectra.

This paper offers two new directions to shape-from-shading, namely the use of the heat equation to smooth the field of surface normals and the recovery of surface height using a low-dimensional embedding. Turning our attention to the first of these contributions, we pose the problem of surface normal recovery as that of solving the steady state heat equation subject to the hard constraint that Lambert's law is satisfied. We perform our analysis on a plane perpendicular to the light source direction, where the z component of the surface normal is equal to the normalized image brightness. The x - y or azimuthal component of the surface normal is found by computing the gradient of a scalar field that evolves with time subject to the heat equation. We solve the heat equation for the scalar potential and, hence, recover the azimuthal component of the surface normal from the average image brightness, making use of a simple finite difference method. The second contribution is to pose the problem of recovering the surface height function as that of embedding the field of surface normals on a manifold so as to preserve the pattern of surface height differences and the lattice footprint of the surface normals. We experiment with the resulting method on a variety of real-world image data, where it produces qualitatively good reconstructed surfaces.

In this paper, we present an enhanced kernel-based tracker for monochromatic and thermographic video. The technique presented here employs the image intensity and the Local Binary Pattern (LBP) to construct a two dimensional histogram representative of the grayscale values and the texture of the target under study. With the histogram at hand, we proceed to compute its power density function. The new location of the object is then determined making use of a mean-shift optimisation approach. We illustrate the performance of our method in both, thermographic and monochromatic footages and compare our results to an alternative.

This paper describes a new robust regular polygon detector. The regular polygon transform is posed as a mixture of regular polygons in a five dimensional space. Given the edge structure of an image, we derive the a posteriori probability for a mixture of regular polygons, and thus the probability density function for the appearance of a mixture of regular polygons. Likely regular polygons can be isolated quickly by discretising and collapsing the search space into three dimensions. The remaining dimensions may be efficiently recovered subsequently using maximum likelihood at the locations of the most likely polygons in the subspace. This leads to an efficient algorithm. Also the a posteriori formulation facilitates inclusion of additional a priori information leading to real-time application to road sign detection. The use of gradient information also reduces noise compared to existing approaches such as the generalised Hough transform. Results are presented for images with noise to show stability. The detector is also applied to two separate applications: real-time road sign detection for on-line driver assistance; and feature detection, recovering stable features in rectilinear environments.

This paper is concerned with computing graph edit distance. One of the criticisms that can be leveled at existing methods for computing graph edit distance is that they lack some of the formality and rigor of the computation of string edit distance. Hence, our aim is to convert graphs to string sequences so that string matching techniques can be used. To do this, we use a graph spectral seriation method to convert the adjacency matrix into a string or sequence order. We show how the serial ordering can be established using the leading eigenvector of the graph adjacency matrix. We pose the problem of graph-matching as a maximum a posteriori probability (MAP) alignment of the seriation sequences for pairs of graphs. This treatment leads to an expression in which the edit cost is the negative logarithm of the a posteriori sequence alignment probability. We compute the edit distance by finding the sequence of string edit operations which minimizes the cost of the path traversing the edit lattice. The edit costs are determined by the components of the leading eigenvectors of the adjacency matrix and by the edge densities of the graphs being matched. We demonstrate the utility of the edit distance on a number of graph clustering problems.

In this paper, we describe a method for correcting underexposed images by recovering the Lambertian diffuse component in the scene. The method makes use of an implicit mapping between the objects in the scene and a unit sphere. As a result of this treatment, the scene radiance function can be represented by a polar function on the unit sphere. We pose the problem of recovering the scene radiance function as that of estimating a tabular representation of this polar function. We demonstrate how image gradient information can be used to perform the required mapping. With the mapping at hand, we generate an image corresponding to the diffuse component of the scene. The diffuse and input images are then blended in order to obtain the corrected image. We present results on images of real-world scenes and provide comparison with an alternative. 1.

Single image facial view synthesis using SFS

We present a parameter-free method for estimating the BRDF of a subject's skin from a single image. We show how the technique can be used to remove specularities caused by perspiration or oil on the skin's surface and demonstrate that this yields improved analysis using shape from shading.

This paper shows how the eigenstructure of the adjacency matrix can be used for the purposes of robust graph matching. We commence from the observation that the leading eigenvector of a transition probability matrix is the steady state of the associated Markov chain. When the transition matrix is the normalized adjacency matrix of a graph, then the leading eigenvector gives the sequence of nodes of the steady state random walk on the graph. We use this property to convert the nodes in a graph into a string where the node-order is given by the sequence of nodes visited in the random walk. We match graphs represented in this way, by finding the sequence of string edit operations which minimize edit distance.

In this paper, we explore how graph-spectral methods can be used to develop a new shape-from-shading algorithm. We characterize the field of surface normals using a weight matrix whose elements are computed from the sectional curvature between different image locations and penalize large changes in surface normal direction. Modeling the blocks of the weight matrix as distinct surface patches, we use a graph seriation method to find a surface integration path that maximizes the sum of curvature-dependent weights and that can be used for the purposes of height reconstruction. To smooth the reconstructed surface, we fit quadrics to the height data for each patch. The smoothed surface normal directions are updated ensuring compliance with Lambert's law. The processes of height recovery and surface normal adjustment are interleaved and iterated until a stable surface is obtained. We provide results on synthetic and real-world imagery.

We are concerned with computing graph edit distance. One of the criticisms that can be leveled at existing methods for computing graph edit distance is that it lacks the formality and rigour of the computation of string edit distance. Hence, our aim is to convert graphs to string sequences so that standard string edit distance techniques can be used. To do this we use graph spectral seriation method to convert the adjacency matrix into a string or sequence order. We pose the problem of graph-matching as maximum a posteriori probability alignment of the seriation sequences for pairs of graphs. This treatment leads to an expression for the edit costs. We compute the edit distance by finding the sequence of string edit operations, which minimise the cost of the path traversing the edit lattice. The edit costs are defined in terms of the a posteriori probability of visiting a site on the lattice. We demonstrate the method with results on a data-set of Delaunay graphs.

In this paper we show how a system for performing automatic surface model acquisition from single object views can be designed. The surface acquisition process is a two step one. Firstly, the surface normals are computed using a shape-from-shading algorithm. Secondly, the field of surface normals is integrated into a 3D surface. For the surface integration step, we have performed experiments with two alternatives. The first of these is a geometric surface integration algorithm. The second alternative comprises a graph-spectral surface integration algorithm. We present results on images of classical statues and provide a preliminary quantitative study.

A Mumford-Shah Diffusion Process for Shape-from-Shading

This paper describes a graph-spectral method for shape-from-shading. The algorithm takes as its input a field of initial surface normal estimates computed using the Lambertian irradiance cone and the Canny edge gradient. We illustrate how to refine this field of surface normals using a graph-spectral method. From the initial field of surface normals we make curvature estimates. The curvature estimates are in turn used to compute a transition probability matrix. From the theory of random walks on graphs, the leading eigenvector of this matrix is a curvature minimising path through the field of surface normals. We impose curvature consistency on the initially noisy field of surface normals by rotating them about their irradiance cones so that they follow the path defined by the leading eigenvector. This method is applied to a variety of real-world images and is shown to lead to improved surface reconstruction.

An EM-like Algorithm for Motion Segmentation via Eigendecomposition

Graph Matching using Adjacency Matrix Markov Chains

The paper describes a probabilistic framework for graph clustering. We commence from a set of pairwise distances between graph structures. From this set of distances, we use a mixture model to characterize the pairwise affinity of the different graphs. We present an EM-like algorithm for clustering the graphs by iteratively updating the elements of the affinity matrix. In the M-step we apply eigendcomposition to the affinity matrix to locate the principal clusters. In the M-step we update the affinity probabilities. We apply the resulting unsupervised clustering algorithm to two practical problems. The first of these involves locating shape-categories using shock trees extracted from 2D silhouettes. The second problem involves finding the view structure of a polyhedral object using the Delaunay triangulation of corner features.

The paper describes a graph-spectral method for 3D surface integration. The algorithm takes as its input a 2D field of surface normal estimates, delivered, for instance, by a shape-from-shading or shape-from-texture procedure. The method borrows ideas from routing theory. We exploit the well-known fact that the leading eigenvector of a Markov chain transition probability matrix is the steady-state random walk on the equivalent weighted graph. We use this property to find the minimum total curvature path through the available surface normals. To do this, we construct a transition probability matrix whose elements are related to the differences in surface normal direction. By threading the surface normals together along the path specified by the magnitude order of the components of the leading eigenvector, we perform surface integration. The height increments along the path are simply related to the traversed path length and the slope of the local tangent plane. The method is evaluated on data delivered by a shape-from-shading algorithm.

This paper investigates whether meaningful shape categories can be identified in an unsupervised way by clustering shock-trees. We commence by computing weighted and unweighted edit distances between shock-trees extracted from the Hamilton-Jacobi skeleton of 2D binary shapes. Next we use an EM-like algorithm to locate pairwise clusters in the pattern of edit-distances. We show that when the tree edit distance is weighted using the geometry of the skeleton, then the clustering method returns meaningful shape categories.

This paper applies a new clustering approach for identifying and segmenting motion in image sequences. We estimate a matrix whose entries represent similarity probabilities between local motion estimates. We adopt a two step iterative algorithm which consists of a variant of the expectation maximization algorithm for segmenting regions with similar motion. The proposed algorithm updates cluster memberships in one step while it maximizes the expected log-likelihood in the second step. The performance of the algorithm is improved greatly by the use of modal sharpening.

This paper presents an eigenclustering approach to line-segment grouping. We make three contributions. First, we show how the geometry of the line-endpoints can be used to compute a grouping field by interpolating a polar lemniscate between them. Second, we show how to adaptively threshold the grouping field to produce a line-adjacency matrix. Finally, we present a non-iterative method for locating line-groupings using the eigenvectors of the adjacency matrix. 1