This work aims to boost the performance of 3D human pose estimators trained in a weakly-supervised setting where there are much fewer annotated 3D poses than unlabeled video data. We formulate two self-supervised pose prior regularizers (PPR) - bone proportion and joint mobility constraints that are pose translation, scale, and rotation invariant. These regularizers, combined with bone symmetry loss, reduce overfitting to the 2D reprojection loss commonly used in weakly-supervised settings by optimizing the bone lengths and joint rotations of estimated 3D poses. Consequently, improving the accuracy of 3D pose estimators. The regularizers are network independent and can be applied to any network architecture without modifications. We apply our proposed PPR to VideoPose3D network [1] and show that it decreases the MPJPE by 24% when using ≤5% of annotated H36M [2] 3D data, improving state-of-the-art accuracy by 7.9 mm.

Deep learning approaches for semantic segmentation rely primarily on supervised learning approaches and require substantial efforts in producing pixel-level annotations. Further, such approaches may perform poorly when applied to unseen image domains. To cope with these limitations, both unsupervised domain adaptation (UDA) with full source supervision but without target supervision and semi-supervised learning (SSL) with partial supervision have been proposed. While such methods are effective at aligning different feature distributions, there is still a need to efficiently exploit unlabeled data to address the performance gap with respect to fully-supervised methods. In this paper we address semi-supervised domain adaptation (SSDA) for semantic segmentation, where a large amount of labeled source data as well as a small amount of labeled target data are available. We propose a novel and effective two-step semi-supervised dual-domain adaptation (SSDDA) approach to address both cross-and intra-domain gaps in semantic segmentation. The proposed framework is comprised of two mixing modules. First, we conduct a cross-domain adaptation via an image-level mixing strategy, which learns to align the distribution shift of features between the source data and target data. Second, intra-domain adaptation is achieved using a separate student-teacher network which is built to generate category-level data augmentation by mixing unlabeled target data in a way that respects predicted object boundaries. We demonstrate that the proposed approach outperforms state-of-the-art methods on two common synthetic-to-real semantic segmentation benchmarks. An extensive ablation study is provided to further validate the effectiveness of our approach.

Data scarcity is a fundamental obstacle to the adoption of deep learning models for a variety of detection tasks in medical images due to inherent difficulties in acquiring and labeling such images. This is particularly so for cerebral microbleed (CMB) detection in community cohorts where datasets are small and targets are hard to find. While numerous methods have been proposed for leveraging unlabeled data and semi-supervised learning to improve detection performance, unlabeled data may not be available in some domains. Data synthesis provides an alternative approach for increasing the size of the dataset but learning a realistic synthesis model using state-of-the-art methods such as generative adversarial networks suffers from the same data scarcity challenges as the detection model. In this work, we propose an end to end approach for refining synthesized examples developed using a priori domain knowledge, in order to improve detection performance using limited amounts of actual data. We combine detection and synthesis in a single network and incorporate detection loss into the synthesis refinement block in order to encourage the generation of realistic synthetic data, while concurrently updating the detection model trained with a small set of real data to enforce realistic synthesis and prevent overfitting. We demonstrate the effectiveness of our approach to improve CMB detection on postmortem MRI data and show significant improvements in average precision over baseline data synthesis and adversarial approaches.

Data for supervised learning of ego-motion and depth from video is scarce and expensive to produce. Subsequently, recent work has focused on unsupervised learning methods and achieved remarkable results. Many unsupervised approaches rely on single-view predicted depth and so ignore motion information. Some unsupervised methods incorporate motion information indirectly by designing the depth prediction network as an RNN. However, none of the existing methods make direct use of multiple frames when predicting depth, which are readily available in videos. In this work, we show that it is possible to achieve superior pose prediction results by modeling motion more directly. Our method uses a novel learning-based formulation for depth propagation and refinement which warps predicted depth maps forward from the current frame onto the next frame where it serves as a prior for predicting the next frame’s depth map. Code will be made available upon acceptance.

Semantic segmentation using convolutional neural networks (CNN) is a crucial component in image analysis. Training a CNN to perform semantic segmentation requires a large amount of labeled data, where the production of such labeled data is both costly and labor intensive. Semi-supervised learning algorithms address this issue by utilizing unlabeled data and so reduce the amount of labeled data needed for training. In particular, data augmentation techniques such as CutMix and ClassMix generate additional training data from existing labeled data. In this paper we propose a new approach for data augmentation, termed ComplexMix, which incorporates aspects of CutMix and ClassMix with improved performance. The proposed approach has the ability to control the complexity of the augmented data while attempting to be semantically-correct and address the tradeoff between complexity and correctness. The proposed ComplexMix approach is evaluated on a standard dataset for semantic segmentation and compared to other state-of-the-art techniques. Experimental results show that our method yields improvement over state-of-the-art methods on standard datasets for semantic image segmentation.

Optimizing the discriminator in Generative Adversarial Networks (GANs) to completion in the inner training loop is computationally prohibitive, and on finite datasets would result in overfitting. To address this, a common update strategy is to alternate between k optimization steps for the discriminator D and one optimization step for the generator G. This strategy is repeated in various GAN algorithms where k is selected empirically. In this paper, we show that this update strategy is not optimal in terms of accuracy and convergence speed, and propose a new update strategy for networks with Wasserstein GAN (WGAN) group related loss functions (e.g. WGAN, WGAN-GP, Deblur GAN, and Super resolution GAN). The proposed update strategy is based on a loss change ratio comparison of G and D. We demonstrate that the proposed strategy improves both convergence speed and accuracy.

In recent years, the emphasis of scientific inquiry has shifted from whole-genome analyses to an understanding of cellular responses specific to tissue, developmental stage or environmental conditions. One of the central mechanisms underlying the diversity and adaptability of the contextual responses is alternative splicing (AS). It enables a single gene to encode multiple isoforms with distinct biological functions. However, to date, the functions of the vast majority of differentially spliced protein isoforms are not known. Integration of genomic, proteomic, functional, phenotypic and contextual information is essential for supporting isoform-based modeling and analysis. Such integrative proteogenomics approaches promise to provide insights into the functions of the alternatively spliced protein isoforms and provide high-confidence hypotheses to be validated experimentally. This manuscript provides a survey of the public databases supporting isoform-based biology. It also presents an overview of the potential global impact of AS on the human canonical gene functions, molecular interactions and cellular pathways.

Change detection is an important task in computer vision and video processing. Due to unimportant or nuisance forms of change, traditional methods require sophisticated image preprocessing and possibly manual interaction. In this work, we propose an end-to-end approach for change detection to identify temporal changes in multiple images. Our approach feeds a pair of images into a deep convolutional neural network combining the model of MatchNet [1] and the Fully Convolutional Network [2] modified to reduce the number of parameters. We train and evaluate the proposed approach using a subset of frames from the Change Detection challenge 2014 dataset (CDnet 2014). Experimental evaluation comparing the performance of the proposed approach with several known approaches shows that the proposed approach outperforms existing methods.

Generating images from word descriptions is a challenging task. Generative adversarial networks(GANs) are shown to be able to generate realistic images of real-life objects. In this paper, we propose a new neural network architecture of LSTM Conditional Generative Adversarial Networks to generate images of real-life objects. Our proposed model is trained on the Oxford-102 Flowers and Caltech-UCSD Birds-200-2011 datasets. We demonstrate that our proposed model produces the better results surpassing other state-of-art approaches.

Using a layered representation for motion estimation has the advantage of being able to cope with discontinuities and occlusions. In this paper, we learn to estimate optical flow by combining a layered motion representation with deep learning. Instead of pre-segmenting the image to layers, the proposed approach automatically generates a layered representation of optical flow using the proposed soft-mask module. The essential components of the soft-mask module are maxout and fuse operations, which enable a disjoint layered representation of optical flow and more accurate flow estimation. We show that by using masks the motion estimate results in a quadratic function of input features in the output layer. The proposed soft-mask module can be added to any existing optical flow estimation networks by replacing their flow output layer. In this work, we use FlowNet as the base network to which we add the soft-mask module. The resulting network is tested on three well-known benchmarks with both supervised and unsupervised flow estimation tasks. Evaluation results show that the proposed network achieve better results compared with the original FlowNet.

This paper presents a novel approach for lecture video indexing using a boosted deep convolutional neural network system. The indexing is performed by matching high quality slide images, for which text is either known or extracted, to lower resolution video frames with possible noise, perspective distortion, and occlusions. We propose a deep neural network integrated with a boosting framework composed of two sub-networks targeting feature extraction and similarity determination to perform the matching. The trained network is given as input a pair of slide image and a candidate video frame image and produces the similarity between them. A boosting framework is integrated into our proposed network during the training process. Experimental results show that the proposed approach is much more capable of handling occlusion, spatial transformations, and other types of noises when compared with known approaches.

Handling imbalanced datasets is a challenging problem that if not treated correctly results in reduced classification performance. Imbalanced datasets are commonly handled using minority oversampling, whereas the SMOTE algorithm is a successful oversampling algorithm with numerous extensions. SMOTE extensions do not have a theoretical guarantee during training to work better than SMOTE and in many instances their performance is data dependent. In this paper we propose a novel extension to the SMOTE algorithm with a theoretical guarantee for improved classification performance. The proposed approach considers the classification performance of both the majority and minority classes. In the proposed approach CGMOS (Certainty Guided Minority OverSampling) new data points are added by considering certainty changes in the dataset. The paper provides a proof that the proposed algorithm is guaranteed to work better than SMOTE for training data. Further, experimental results on 30 real-world datasets show that CGMOS works better than existing algorithms when using 6 different classifiers.

Learning from synthetic data has many important and practical applications, An example of application is photo-sketch recognition. Using synthetic data is challenging due to the differences in feature distributions between synthetic and real data, a phenomenon we term synthetic gap. In this paper, we investigate and formalize a general framework -- Stacked Multichannel Autoencoder (SMCAE) that enables bridging the synthetic gap and learning from synthetic data more efficiently. In particular, we show that our SMCAE can not only transform and use synthetic data on the challenging face-sketch recognition task, but that it can also help simulate real images, which can be used for training classifiers for recognition. Preliminary experiments validate the effectiveness of the framework.

Automatic roof style classification using point clouds is useful and can be used as a prior knowledge in various applications, such as the construction of 3D models of real-world buildings. Previous classification approaches usually employ heuristic rules to recognize roof style and are limited to a few roof styles. In this paper, the recognition of roof style is done by a roof style classifier which is trained based on bag of words features extracted from a point cloud. In the computation of bag of words features, a key challenge is the generation of the codebook. Unsupervised learning is often misguided easily by the data and detects uninteresting patterns within the data. In contrast, we propose to integrate existing knowledge of roof structure and cluster the points of target roof styles into several semantic classes which can then be used as code words in the bag of words model. We use synthetic variants of these code words to train a semantics point classifier. We evaluate our approach on two datasets with different levels of degradations. We compare the results of our approach with two unsupervised learning algorithms: K-Means and Gaussian Mixture Model. We show that our approach achieve higher accuracy in classification of the roof styles and maintains consistent performance among different datasets.

In recent years due to the rise of social, biological, and other rich content graphs, several new graph clustering methods using structure and node's attributes have been introduced. In this paper, we proposed an effective benchmark to evaluate these new methods. Our benchmark is an attributes extension to a widely used structure only benchmark. We also developed a new clustering method, termed Selection method, that uses the graph structure ambiguity to switch between structure and attribute clustering methods. Using the new benchmark and Normalized Mutual Information (NMI) metric, we evaluated the Selection method against five clustering methods: three structure and attribute methods, one structure only method and one attribute only method. We showed that the Selection method outperformed that state-of-art structure and attribute methods.

Large degradations in document images impede their readability and deteriorate the performance of automated document processing systems. Document image quality (IQ) metrics have been defined through optical character recognition (OCR) accuracy. Such metrics, however, do not always correlate with human perception of IQ. When enhancing document images with the goal of improving readability, e.g., in historical documents where OCR performance is low and/or where it is necessary to preserve the original context, it is important to understand human perception of quality. The goal of this paper is to design a system that enables the learning and estimation of human perception of document IQ. Such a metric can be used to compare existing document enhancement methods and guide automated document enhancement. Moreover, the proposed methodology is designed as a general framework that can be applied in a wide range of applications.

Warping is fundamental to multiple algorithms in computer vision and medical imaging such as image and volume registration. Warping is performed by determining a continuous deformation map and applying it to a given image or volume. In registration the deformation map is determined based on correspondence between two images. It is often the case that the deformation map can only be determined at discrete locations and so has to be interpolated. The discrete locations where the deformation map is determined form irregular sampling of the unknown continuous deformation map. Thin-plate splines are commonly used to perform the interpolation and provide an optimal solution in the sense of bending energy minimization. Assuming N samples of the deformation map and n2image pixels, thin plate splines require solving a N × N dense linear system with O(N3) complexity for determining spline coefficients and N computations per pixel with O(Nn2) complexity for determining interpolated values. When N and n are large as in the case of volumetric medical image analysis this cost becomes prohibitive. The approach proposed in this paper is based on subdivision surfaces and is capable of achieving similar quality results with O (N log N) complexity for co efficient determination and O(n2) complexity for computing interpolated values. Experimental results demonstrate two orders of magnitude performance improvement on actual clinical data.

Markov random field (MRF) based approaches have been shown to perform well in a wide range of applications. Due to the iterative nature of the algorithm, the computational cost of such applications is normally high. In the context of document image analysis, where numerous documents have to be processed, this computational cost may become prohibitive. We describe a novel approach to document image enhancement using MRF.We show that by using domain specific knowledge, we are able to substantially improve computational performance by an order of magnitude. Moreover, in contrast to known techniques where patch initialization is arbitrary, in the proposed approach patch initialization is data consistent and so results in improved effectiveness. Experimental results comparing the proposed approach to known techniques using historical documents from the Frieder Collection are provided.

Shape matching is fundamental to numerous computer vision algorithms and may be used for similarity determination and registration. Establishing correspondence and measuring similarity between shapes is of great importance. Shape matching often involves simultaneous estimation of both a correspondence and an alignment transformation. Such an estimate is particularly difficult when the alignment transformation is non-linear and so contains a large number of degrees of freedom. We describe a novel approach for shape matching that is based on shape contexts and uses particle dynamics warping to maximize the similarity of shapes while satisfying structural constraints. The approach is based on an iterative solution of a system of first order ordinary differential equations. The main advantage of the proposed approach is its ability to incorporate shape constraints into the matching process. Furthermore, the proposed approach does not require a solution of an optimal assignment problem which is sensitive to outliers, and does not require thin-plate spline warping which is computationally expensive. To illustrate the applicability of our approach we address the problem of offline signature recognition which in contrast to online signature recognition does not provide for a simple parametrization of the signature curves. The proposed approach is evaluated by measuring the precision and recall rates of documents based on signature similarity. To facilitate a realistic evaluation, the signature data we use was collected from real world documents spanning a period of several decades.

Offline signature recognition is an important form of biometric identification that can be used for various purposes. Similar to other biometric measures, signatures have inherent variability and so pose a difficult recognition problem. In this paper, we explore a novel approach for reducing the variability associated with matching signatures based on curve warping. Existing techniques, such as the dynamic time warping approach, address this problem by minimizing a cost function through dynamic programming. This is by nature a 1-D optimization process that is possible when a 1-D parametrization of the curves is known. In this paper, we propose a novel approach for solving the curve correspondence problem that is not limited by the requirement of 1-D parametrization. The proposed approach utilizes particle dynamics and minimizes a cost function through an iterative solution of a system of first-order ordinary differential equations. The proposed approach is, therefore, capable of handling complex curves for which a simple parametrization is not available. The proposed approach is evaluated by measuring the precision and recall rates of documents based on signature similarity. To facilitate a realistic evaluation, the signature data we use were collected from real-world documents, spanning a period of several decades.

Lesion segmentation in MRI scans is used for lesion quantification as pertaining to various medical conditions. We propose a novel technique for chronic stroke lesion segmentation based on multiple modalities including T1-weighted and T2-weighted images as well as diffusion tensor-based modalities. The proposed approach is based on a mixture-parametric probabilistic model whereas the model parameters are optimized by maximizing the incomplete-data log-likelihood function through expectation maximization. The mixture components are selected to have Cauchy distributions thus facilitating efficient computation and increased robustness to noise. A probabilistic prior is computed by evaluating the feature vectors for a set of registered brain scans in a control set. Experimental results on actual clinical data demonstrate the effectiveness of the proposed approach.

We developed a prototype for integrated retrieval and aggregation of diverse information contained in scanned paper documents. Such complex document information processing combines several forms of image processing together with textual/linguistic processing to enable effective analysis of complex document collections, a necessity for a wide range of applications. This is the first system to attempt integrated retrieval from complex documents; we report its current capabilities.

Research and development of information access technology for scanned paper documents has been hampered by the lack of public test collections of realistic scope and complexity. As part of a project to create a prototype system for search and mining of masses of document images, we are assembling a 1.5 terabyte dataset to support evaluation of both end-to-end complex document information processing (CDIP) tasks (e.g., text retrieval and data mining) as well as component technologies such as optical character recognition (OCR), document structure analysis, signature matching, and authorship attribution.

Vessel enhancement in volumetric data is a necessary prerequisite in various medical imaging applications with particular importance for automated nodule detection. Ideally, vessel enhancement filters should enhance vessels and vessel junctions while suppressing nodules and other non-vessel elements. A distinction between vessels and nodules is normally obtained through eigenvalue analysis of the curvature tensor which is a second order differential quantity and so is sensitive to noise. Furthermore, by relying on principal curvatures alone, existing vessel enhancement filters are incapable of distinguishing between nodules and vessel junctions. In this paper we propose probabilistic vessel models from which novel vessel enhancement filters capable of enhancing junctions while suppressing nodules are derived. The proposed filters are based on eigenvalue analysis of the structure tensor which is a first order differential quantity and so are less sensitive to noise. The proposed filters are evaluated and compared to known techniques based on actual clinical data.

Vessel tree reconstruction in volumetric data is a necessary prerequisite in various medical imaging applications. Specifically, when considering the application of automated lung nodule detection in thoracic computed tomography (CT) scans, vessel trees can be used to resolve local ambiguities based on global considerations and so improve the performance of nodule detection algorithms. In this study, a novel approach to vessel tree reconstruction and its application to nodule detection in thoracic CT scans was developed by using correlation-based enhancement filters and a fuzzy shape representation of the data. The proposed correlation-based enhancement filters depend on first-order partial derivatives and so are less sensitive to noise compared with Hessian-based filters. Additionally, multiple sets of eigenvalues are used so that a distinction between nodules and vessel junctions becomes possible. The proposed fuzzy shape representation is based on regulated morphological operations that are less sensitive to noise. Consequently, the vessel tree reconstruction algorithm can accommodate vessel bifurcation and discontinuities. A quantitative performance evaluation of the enhancement filters and of the vessel tree reconstruction algorithm was performed. Moreover, the proposed vessel tree reconstruction algorithm reduced the number of false positives generated by an existing nodule detection algorithm by 38%.

Accurate curvature estimation in discrete surfaces is an important problem with numerous applications. Curvature is an indicator of ridges and can be used in applications such as shape analysis and recognition, object segmentation, adaptive smoothing, anisotropic fairing of irregular meshes, and anisotropic texture mapping. In this paper, a new framework is proposed for accurate curvature estimation in discrete surfaces. The proposed framework is based on a local directional curve sampling of the surface where the sampling frequency can be controlled. This local model has a large number of degrees of freedoms compared with known techniques and, so, can better represent the local geometry. The proposed framework is quantitatively evaluated and compared with common techniques for surface curvature estimation. In order to perform an unbiased evaluation in which smoothing effects are factored out, we use a set of randomly generated Bezier surface patches for which the curvature values can be analytically computed. It is demonstrated that, through the establishment of sampling conditions, the error in estimations obtained by the proposed framework is smaller and that the proposed framework is less sensitive to low sampling density, sampling irregularities, and sampling noise.

A common task in cytogenetic tests is the classification of human chromosomes. Successful separation between touching and overlapping chromosomes in a metaphase image is vital for correct classification. Current systems for automatic chromosome classification are mostly interactive and require human intervention for correct separation between touching and overlapping chromosomes. Since chromosomes are nonrigid objects, special separation methods are required to segregate them. Common methods for overlapping chromosomes separation between touching chromosomes tend to fail where ambiguity or incomplete information are involved, and so are unable to segregate overlapping chromosomes. The proposed approach treats the separation problem as an identification problem, and, in this way, manages to segregate overlapping chromosomes. This approach encompasses low-level knowledge about the objects and uses only extracted information, therefore, it is fast and does not depend on the existence of a separating path. The method described in this paper can be adopted for other applications, where separation between touching and overlapping nonrigid objects is required.

Directional information has a strong meaning in line-drawing images. Therefore, the proposed approach for line-drawing images processing is based on a directional decomposition of the input image into a set of images each containing line segments in a respective range of directions. The directional information is then processed by using newly defined directional morphological operators, called tube-directional, that have accurate selectivity and controllable strictness. When using morphological operators, adaptation to a specific task is achieved only globally by setting the structure of the morphological kernel. The concept of one global adjustment for many local operations is some how conflicting. In the proposed approach the parameters of the tube-directional morphological operators are determined locally for each element of the processed image.

The use of computer aided design requires line drawings and maps to be digitized and stored in databases. The input of line drawings and maps into databases requires vectorization of lines, and recognition of symbols and characters. The paper addresses two aspects related to the input process. The first aspect is an automatic algorithm for the separation of character strings from maps. The second aspect is an algorithm for line thinning. The proposed algorithms are based on directional morphology operations. The character string extraction algorithm is independent of font style, size, and language and is suitable for a variety of map styles with straight or curved lines. The presented experimental results demonstrate very good performance of the algorithms even in cases where the character strings touch or intersect lines in the map.