Inspired by the diversity of biological neurons, quadratic artificial neurons can play an important role in deep learning models. The type of quadratic neurons of our interest replaces the inner-product operation in the conventional neuron with a quadratic function. Despite promising results so far achieved by networks of quadratic neurons, there are important issues not well addressed. Theoretically, the superior expressivity of a quadratic network over either a conventional network or a conventional network via quadratic activation is not fully elucidated, which makes the use of quadratic networks not well grounded. In practice, although a quadratic network can be trained via generic backpropagation, it can be subject to a higher risk of collapse than the conventional counterpart. To address these issues, we first apply the spline theory and a measure from algebraic geometry to give two theorems that demonstrate better model expressivity of a quadratic network than the conventional counterpart with or without quadratic activation. Then, we propose an effective training strategy referred to as referenced linear initialization (ReLinear) to stabilize the training process of a quadratic network, thereby unleashing the full potential in its associated machine learning tasks. Comprehensive experiments on popular datasets are performed to support our findings and confirm the performance of quadratic deep learning. We have shared our code in https://github.com/FengleiFan/ReLinear.

A neural network (NN) with the widely-used ReLU activation has been shown to partition the sample space into many convex polytopes for prediction. However, the parametric way a NN and other machine learning models use to partition the space has imperfections, e.g., the compromised interpretability for complex models, the inflexibility in decision boundary construction due to the generic character of the model, and the risk of being trapped into shortcut solutions. In contrast, although the nonparameterized models can adorably avoid or downplay these issues, they are usually insufficiently powerful either due to over-simplification or the failure to accommodate the manifold structures of data. In this context, we first propose a new type of machine learning models referred to as Manifoldron that directly derives decision boundaries from data and partitions the space via manifold structure discovery. Then, we systematically analyze the key characteristics of the Manifoldron such as manifold characterization capability and its link to NNs. The experimental results on four synthetic examples, 20 public benchmark datasets, and one real-world application demonstrate that the proposed Manifoldron performs competitively compared to the mainstream machine learning models. We have shared our code in https://github.com/wdayang/Manifoldron for free download and evaluation.

While classic studies proved that wide networks allow universal approximation, recent research and successes of deep learning demonstrate the power of deep networks. Based on a symmetric consideration, we investigate if the design of artificial neural networks should have a directional preference, and what the mechanism of interaction is between the width and depth of a network. Inspired by the De Morgan law, we address this fundamental question by establishing a quasi-equivalence between the width and depth of ReLU networks. We formulate two transforms for mapping an arbitrary ReLU network to a wide ReLU network and a deep ReLU network respectively, so that the essentially same capability of the original network can be implemented. Based on our findings, a deep network has a wide equivalent, and vice versa, subject to an arbitrarily small error.

Convolution has played a prominent role in various applications in science and engineering for many years and has become a key operation in many neural networks. There has been a recent growth of interest in generalizing convolutions on three-dimensional surfaces, often represented as compact manifolds. However, existing approaches cannot preserve all the desirable properties of Euclidean convolutions, namely, compactly supported filters, directionality, and transferability across different manifolds. This paper develops a new generalization of the convolution operation, referred to as parallel transport convolution (PTC), on Riemannian manifolds and their discrete counterparts. PTC is designed based on parallel transportation that can translate information along a manifold and intrinsically preserve directionality. Furthermore, PTC allows for the construction of compactly supported filters and is also robust to manifold deformations. This enables us to perform waveletlike operations and to define convolutional neural networks on curved domains.

In this work, a robust and efficient dual iterative refinement (DIR) method is proposed for dense correspondence between two nearly isometric shapes. The key idea is to use dual information, such as spatial and spectral, or local and global features, in a complementary and effective way, and extract more accurate information from current iteration to use for the next iteration. In each DIR iteration, starting from current correspondence, a zoom-in process at each point is used to select well matched anchor pairs by a local mapping distortion criterion. These selected anchor pairs are then used to align spectral features (or other appropriate global features) whose dimension adaptively matches the capacity of the selected anchor pairs. Thanks to the effective combination of complementary information in a data-adaptive way, DIR is not only efficient but also robust to render accurate results within a few iterations. By choosing appropriate dual features, DIR has the flexibility to handle patch and partial matching as well. Our comprehensive experiments on various data sets demonstrate the superiority of DIR over other state-of-the-art methods in terms of both accuracy and efficiency.

Abstract We propose a method to simultaneously compute scalar basis functions with an associated functional map for a given pair of triangle meshes. Unlike previous techniques that put emphasis on smoothness with respect to the Laplace–Beltrami operator and thus favor low‐frequency eigenfunctions, we aim for a basis that allows for better feature matching. This change of perspective introduces many degrees of freedom into the problem allowing to better exploit non‐smooth descriptors. To effectively search in this high‐dimensional space of solutions, we incorporate into our minimization state‐of‐the‐art regularizers. We solve the resulting highly non‐linear and non‐convex problem using an iterative scheme via the Alternating Direction Method of Multipliers. At each step, our optimization involves simple to solve linear or Sylvester‐type equations. In practice, our method performs well in terms of convergence, and we additionally show that it is similar to a provably convergent problem. We show the advantages of our approach by extensively testing it on multiple datasets in a few applications including shape matching, consistent quadrangulation and scalar function transfer.

In this work, we introduce a novel local pairwise descriptor and then develop a simple, effective iterative method to solve the resulting quadratic assignment through sparsity control for shape correspondence between two approximate isometric surfaces. Our pairwise descriptor is based on the stiffness and mass matrix of finite element approximation of the Laplace-Beltrami differential operator, which is local in space, sparse to represent, and extremely easy to compute while containing global information. It allows us to deal with open surfaces, partial matching, and topological perturbations robustly. To solve the resulting quadratic assignment problem efficiently, the two key ideas of our iterative algorithm are: 1) select pairs with good (approximate) correspondence as anchor points, 2) solve a regularized quadratic assignment problem only in the neighborhood of selected anchor points through sparsity control. These two ingredients can improve and increase the number of anchor points quickly while reducing the computation cost in each quadratic assignment iteration significantly. With enough high-quality anchor points, one may use various pointwise global features with reference to these anchor points to further improve the dense shape correspondence. We use various experiments to show the efficiency, quality, and versatility of our method on large data sets, patches, and point clouds (without global meshes).

The loss landscapes of deep neural networks are not well understood due to their high nonconvexity. Empirically, the local minima of these loss functions can be connected by a learned curve in model space, along which the loss remains nearly constant; a feature known as mode connectivity. Yet, current curve finding algorithms do not consider the influence of symmetry in the loss surface created by model weight permutations. We propose a more general framework to investigate the effect of symmetry on landscape connectivity by accounting for the weight permutations of the networks being connected. To approximate the optimal permutation, we introduce an inexpensive heuristic referred to as neuron alignment. Neuron alignment promotes similarity between the distribution of intermediate activations of a model along the curve with that of the endpoint models. We provide theoretical analysis establishing the benefit of alignment to mode connectivity based on this simple heuristic. We empirically verify that the permutation given by alignment is locally optimal via a proximal alternating minimization scheme. Empirically, optimizing the weight permutation is critical for efficiently learning a simple, planar, low-loss curve between networks that successfully generalizes. Our alignment method can significantly alleviate the recently identified robust loss barrier on the path connecting two adversarial robust models and find more robust and accurate models on the path.

The Euclidean distance geometry problem arises in a wide variety of applications, from determining molecular conformations in computational chemistry to localization in sensor networks. When the distance information is incomplete, the problem can be formulated as a nuclear norm minimization problem. In this paper, this minimization program is recast as a matrix completion problem of a low-rank$r$Gram matrix with respect to a suitable basis. The well-known restricted isometry property cannot be satisfied in this scenario. Instead, a dual basis approach is introduced to theoretically analyze the reconstruction problem. If the Gram matrix satisfies certain coherence conditions with parameter$\nu $, the main result shows that the underlying configuration of$n$points can be recovered with very high probability from$O(nr\nu \log ^{2}(n))$uniformly random samples. Computationally, simple and fast algorithms are designed to solve the Euclidean distance geometry problem. Numerical tests on different 3-D data and protein molecules validate the effectiveness and efficiency of the proposed algorithms.

For node level graph encoding, a recent important state-of-art method is the graph convolutional networks (GCN), which nicely integrate local vertex features and graph topology in the spectral domain. However, current studies suffer from several drawbacks: (1) graph CNNs rely on Chebyshev polynomial approximation which results in oscillatory approximation at jump discontinuities; (2) Increasing the order of Chebyshev polynomial can reduce the oscillations issue, but also incurs unaffordable computational cost; (3) Chebyshev polynomials require degree Ω(poly(1/ε)) to approximate a jump signal such as |x|, while rational function only needs O(poly log(1/ε)). However, it is non-trivial to apply rational approximation without increasing computational complexity due to the denominator. In this paper, the superiority of rational approximation is exploited for graph signal recovering. RatioanlNet is proposed to integrate rational function and neural networks. We show that the rational function of eigenvalues can be rewritten as a function of graph Laplacian, which can avoid multiplication by the eigenvector matrix. Focusing on the analysis of approximation on graph convolution operation, a graph signal regression task is formulated. Under graph signal regression task, its time complexity can be significantly reduced by graph Fourier transform. To overcome the local minimum problem of neural networks model, a relaxed Remez algorithm is utilized to initialize the weight parameters. Convergence rate of RatioanlNet and polynomial based methods on a jump signal is analyzed for a theoretical guarantee. The extensive experimental results demonstrated that our approach could effectively characterize the jump discontinuities, outperforming competing methods by a substantial margin on both synthetic and real-world graphs.

In this work, we propose computational models and algorithms for point cloud registration with nonrigid transformation. First, point clouds sampled from manifolds originally embedded in some Euclidean space are transformed to new point clouds embedded in $\mathbb{R}^n$ by the Laplace--Beltrami (LB) eigenmap, which is invariant under isometric transformation, using the first $n$ leading eigenvalues and corresponding eigenfunctions of the LB operator. Then we develop computational models and algorithms for registration of the transformed point clouds in a distribution/probability sense based on optimal transport, which provides both generality and flexibility for point cloud registration. In particular, we propose to use a rotation-invariant sliced-Wasserstein distance to achieve computation efficiency and handle ambiguities introduced by LB eigenmaps. By going from smaller $n$, which provides a quick and robust registration in coarse scale as well as a good initial guess for registration in finer scale, to a larger $n$, our method provides an efficient and robust multiscale nonrigid point cloud registration.

In this paper, we present a novel approach for the intrinsic mapping of anatomical surfaces and its application in brain mapping research. Using the Laplace-Beltrami eigen-system, we represent each surface with an isometry invariant embedding in a high dimensional space. The key idea in our system is that we realize surface deformation in the embedding space via the iterative optimization of a conformal metric without explicitly perturbing the surface or its embedding. By minimizing a distance measure in the embedding space with metric optimization, our method generates a conformal map directly between surfaces with highly uniform metric distortion and the ability of aligning salient geometric features. Besides pairwise surface maps, we also extend the metric optimization approach for group-wise atlas construction and multi-atlas cortical label fusion. In experimental results, we demonstrate the robustness and generality of our method by applying it to map both cortical and hippocampal surfaces in population studies. For cortical labeling, our method achieves excellent performance in a cross-validation experiment with 40 manually labeled surfaces, and successfully models localized brain development in a pediatric study of 80 subjects. For hippocampal mapping, our method produces much more significant results than two popular tools on a multiple sclerosis study of 109 subjects.

A new image decomposition scheme, called the adaptive directional total variation (ADTV) model, is proposed to achieve effective segmentation and enhancement for latent fingerprint images in this work. The proposed model is inspired by the classical total variation models, but it differentiates itself by integrating two unique features of fingerprints; namely, scale and orientation. The proposed ADTV model decomposes a latent fingerprint image into two layers: cartoon and texture. The cartoon layer contains unwanted components (e.g., structured noise) while the texture layer mainly consists of the latent fingerprint. This cartoon-texture decomposition facilitates the process of segmentation, as the region of interest can be easily detected from the texture layer using traditional segmentation methods. The effectiveness of the proposed scheme is validated through experimental results on the entire NIST SD27 latent fingerprint database. The proposed scheme achieves accurate segmentation and enhancement results, leading to improved feature detection and latent matching performance.

In this paper we present a novel system for the automated reconstruction of cortical surfaces from T1-weighted magnetic resonance images. At the core of our system is a unified Reeb analysis framework for the detection and removal of geometric and topological outliers on tissue boundaries. Using intrinsic Reeb analysis, our system can pinpoint the location of spurious branches and topological outliers, and correct them with localized filtering using information from both image intensity distributions and geometric regularity. In this system, we have also developed enhanced tissue classification with Hessian features for improved robustness to image inhomogeneity, and adaptive interpolation to achieve sub-voxel accuracy in reconstructed surfaces. By integrating these novel developments, we have a system that can automatically reconstruct cortical surfaces with improved quality and dramatically reduced computational cost as compared with the popular FreeSurfer software. In our experiments, we demonstrate on 40 simulated MR images and the MR images of 200 subjects from two databases: the Alzheimer’s Disease Neuroimaging Initiative (ADNI) and International Consortium of Brain Mapping (ICBM), the robustness of our method in large scale studies. In comparisons with FreeSurfer, we show that our system is able to generate surfaces that better represent cortical anatomy and produce thickness features with higher statistical power in population studies.

In this paper, we propose a general framework for approximating differential operator directly on point clouds and use it for geometric understanding on them. The discrete approximation of differential operator on the underlying manifold represented by point clouds is based only on local approximation using nearest neighbors, which is simple, efficient and accurate. This allows us to extract the complete local geometry, solve partial differential equations and perform intrinsic calculations on surfaces. Since no mesh or parametrization is needed, our method can work with point clouds in any dimensions or co-dimensions or even with variable dimensions. The computation complexity scaled well with the number of points and the intrinsic dimensions (rather than the embedded dimensions). We use this method to define the Laplace-Beltrami (LB) operator on point clouds, which links local and global information together. With this operator, we propose a few key applications essential to geometric understanding for point clouds, including the computation of LB eigenvalues and eigenfunctions, the extraction of skeletons from point clouds, and the extraction of conformal structures from point clouds.

Latent fingerprint detection and segmentation play a critical role in image forensics for law enforcement. Being collected from crime scenes, a latent fingerprint is often mixed with other components such as structured noise or other fingerprints. Existing fingerprint recognition algorithms fail to work properly for latent fingerprint images, since they are mostly applicable under the assumption that the image is already properly segmented and there is no overlap between the target fingerprint and other components. In this work, we present a novel directional total variation (DTV) model to achieve effective latent fingerprint detection and segmentation. As compared with existing total variation models, the proposed DTV model differentiates itself by considering spatial-dependent texture orientations in the TV computation, which is particularly suitable for images with oriented textures. We demonstrate the superior performance of the proposed DTV technique using images from the NIST SD27 latent fingerprint database.

The level set method is a popular technique for tracking moving interfaces in several disciplines, including computer vision and fluid dynamics. However, despite its high flexibility, the original level set method is limited by two important numerical issues. First, the level set method does not implicitly preserve the level set function as a distance function, which is necessary to estimate accurately geometric features, s.a. the curvature or the contour normal. Second, the level set algorithm is slow because the time step is limited by the standard Courant-Friedrichs-Lewy (CFL) condition, which is also essential to the numerical stability of the iterative scheme. Recent advances with graph cut methods and continuous convex relaxation methods provide powerful alternatives to the level set method for image processing problems because they are fast, accurate, and guaranteed to find the global minimizer independently to the initialization. These recent techniques use binary functions to represent the contour rather than distance functions, which are usually considered for the level set method. However, the binary function cannot provide the distance information, which can be essential for some applications, s.a. the surface reconstruction problem from scattered points and the cortex segmentation problem in medical imaging. In this paper, we propose a fast algorithm to preserve distance functions in level set methods. Our algorithm is inspired by recent efficient l(1) optimization techniques, which will provide an efficient and easy to implement algorithm. It is interesting to note that our algorithm is not limited by the CFL condition and it naturally preserves the level set function as a distance function during the evolution, which avoids the classical re-distancing problem in level set methods. We apply the proposed algorithm to carry out image segmentation, where our methods prove to be 5-6 times faster than standard distance preserving level set techniques. We also present two applications where preserving a distance function is essential. Nonetheless, our method stays generic and can be applied to any level set methods that require the distance information.

Corpus callosum (CC) is an important structure in human brain anatomy. In this work, we propose a fully automated and robust approach to extract corpus callosum from T1-weighted structural MR images. The novelty of our method is composed of two key steps. In the first step, we find an initial guess for the curve representation of CC by using the zero level set of the first nontrivial Laplace-Beltrami (LB) eigenfunction on the white matter surface. In the second step, the initial curve is deformed toward the final solution with a geodesic curvature flow on the white matter surface. For numerical solution of the geodesic curvature flow on surfaces, we represent the contour implicitly on a triangular mesh and develop efficient numerical schemes based on finite element method. Because our method depends only on the intrinsic geometry of the white matter surface, it is robust to orientation differences of the brain across population. In our experiments, we validate the proposed algorithm on 32 brains from a clinical study of multiple sclerosis disease and demonstrate that the accuracy of our results.

For various 3D shape analysis tasks, the Laplace-Beltrami(LB) embedding has become increasingly popular as it enables the efficient comparison of shapes based on intrinsic geometry. One fundamental difficulty in using the LB embedding, however, is the ambiguity in the eigen-system, and it is conventionally only handled in a heuristic way. In this work, we propose a novel and intrinsic metric, the spectral l2-distance, to overcome this difficulty. We prove mathematically that this new distance satisfies the conditions of a rigorous metric. Using the resulting optimal embedding determined by the spectral l2-distance, we can perform both local and global shape analysis intrinsically in the embedding space. We demonstrate this by developing a template matching approach in the optimal embedding space to solve the challenging problem of identifying major sulci on vervet cortical surfaces. In our experiments, we validate the robustness of our method by the successful identification of major sulcal lines on a large data set of 698 cortical surfaces and illustrate its potential in brain mapping studies.

In medical shape analysis, a critical problem is reconstructing a smooth surface of correct topology from a binary mask that typically has spurious features due to segmentation artifacts. The challenge is the robust removal of these outliers without affecting the accuracy of other parts of the boundary. In this paper, we propose a novel approach for this problem based on the Laplace-Beltrami (LB) eigen-projection and properly designed boundary deformations. Using the metric distortion during the LB eigen-projection, our method automatically detects the location of outliers and feeds this information to a well-composed and topology-preserving deformation. By iterating between these two steps of outlier detection and boundary deformation, we can robustly filter out the outliers without moving the smooth part of the boundary. The final surface is the eigen-projection of the filtered mask boundary that has the correct topology, desired accuracy and smoothness. In our experiments, we illustrate the robustness of our method on different input masks of the same structure, and compare with the popular SPHARM tool and the topology preserving level set method to show that our method can reconstruct accurate surface representations without introducing artificial oscillations. We also successfully validate our method on a large data set of more than 900 hippocampal masks and demonstrate that the reconstructed surfaces retain volume information accurately.