Protecting data privacy in deep learning (DL) is of crucial importance. Several celebrated privacy notions have been established and used for privacy-preserving DL. However, many existing mechanisms achieve privacy at the cost of significant utility degradation and computational overhead. In this paper, we propose a stochastic differential equation-based residual perturbation for privacy-preserving DL, which injects Gaussian noise into each residual mapping of ResNets. Theoretically, we prove that residual perturbation guarantees differential privacy (DP) and reduces the generalization gap of DL. Empirically, we show that residual perturbation is computationally efficient and outperforms the state-of-the-art differentially private stochastic gradient descent (DPSGD) in utility maintenance without sacrificing membership privacy.

Abstract Unoriented surface reconstruction is an important task in computer graphics. Recently, methods based on the Gauss formula or winding number have achieved state‐of‐the‐art performance in both orientation and surface reconstruction. The Gauss formula or winding number, derived from the fundamental solution of the Laplace equation, initially found applications in calculating potentials in electromagnetism. Inspired by the practical necessity of calculating potentials in diverse electromagnetic media, we consider the anisotropic Laplace equation to derive the anisotropic Gauss formula and apply it to surface reconstruction, called “anisotropic Gauss reconstruction”. By leveraging the flexibility of anisotropic coefficients, additional constraints can be introduced to the indicator function. This results in a stable linear system, eliminating the need for any artificial regularization. In addition, the oriented normals can be refined by computing the gradient of the indicator function, ultimately producing high‐quality normals and surfaces. Regarding the space/time complexity, we propose an octree‐based acceleration algorithm to achieve a space complexity of O(N) and a time complexity of O(NlogN). Our method can reconstruct ultra‐large‐scale models (exceeding 5 million points) within 4 minutes on an NVIDIA RTX 4090 GPU. Extensive experiments demonstrate that our method achieves state‐of‐the‐art performance in both orientation and reconstruction, particularly for models with thin structures, small holes, or high genus. Both CuPy‐based and CUDA‐accelerated implementations are made publicly available at https://github.com/mayueji/AGR .

Differential equations have demonstrated intrinsic connections to network structures, linking discrete network layers through continuous equations. Most existing approaches focus on the interaction between ordinary differential equations (ODEs) and feature transformations, primarily working on input signals. In this paper, we study the partial differential equation (PDE) model of neural networks, viewing the neural network as a functional operating on a base model provided by the last layer of the classifier. Inspired by scale-space theory, we theoretically prove that this mapping can be formulated by a convection-diffusion equation, under interpretable and intuitive assumptions from both neural network and PDE perspectives. This theoretically certified framework covers various existing network structures and training techniques, offering a mathematical foundation and new insights into neural networks. Moreover, based on the convection-diffusion equation model, we design a new network structure that incorporates a diffusion mechanism into the network architecture from a PDE perspective. Extensive experiments on benchmark datasets and real-world applications confirm the effectiveness of the proposed model.

Unoriented surface reconstructions based on the Gauss formula have attracted much attention due to their mathematical formulation and good experimental performance. However, the formula’s isotropy limits its capacity to leverage the directional features of point clouds. In this study, we introduce a convection augmentation term to extend the classic Gauss formula. This new term allows our method to leverage point clouds’ directional characteristics effectively. With the proper choice of the velocity field, this method could construct more equations to calculate a more precise indicator function. Furthermore, an adaptive selection strategy of the velocity field is proposed. For large-scale point clouds, we propose a CUDA-and-octree-based acceleration algorithm with O(N) space complexity and O(N log N) time complexity. Our method can complete the orientation and reconstruction tasks of point clouds with up to 500K within a few seconds. Extensive experiments demonstrate that our method achieves state-of-the-art performance and manages various challenging situations, especially for models with thin structures or small holes. The source code is publicly available at https://github.com/mayueji/CAGR .

Diffusion, a fundamental internal mechanism emerging in many physical processes, describes the interaction among different objects. In many learning tasks with limited training samples, the diffusion connects the labeled and unlabeled data points and is a critical component for achieving high classification accuracy. Many existing deep learning approaches directly impose the fusion loss when training neural networks. In this work, inspired by the convection-diffusion ordinary differential equations (ODEs), we propose a novel diffusion residual network (Diff-ResNet), internally introduces diffusion into the architectures of neural networks. Under the structured data assumption, it is proved that the proposed diffusion block can increase the distance-diameter ratio that improves the separability of inter-class points and reduces the distance among local intra-class points. Moreover, this property can be easily adopted by the residual networks for constructing the separable hyperplanes. Extensive experiments of synthetic binary classification, semi-supervised graph node classification and few-shot image classification in various datasets validate the effectiveness of the proposed method.

Estimating consistently oriented normals for point clouds enables a number of important applications in computer graphics such as surface reconstruction. While local normal estimation is possible with simple techniques like principal component analysis (PCA), orienting these normals to be globally consistent has been a notoriously difficult problem. Some recent methods exploit various properties of the winding number formula to achieve global consistency with state-of-the-art performance. Despite their exciting progress, these algorithms either have high space/time complexity, or do not produce accurate and consistently oriented normals for imperfect data. In this paper, we propose a novel property from the winding number formula, termed Winding Number Normal Consistency (WNNC ), to tackle this problem. The derived property is based on the simple observation that the normals (negative gradients) sampled from the winding number field should be codirectional to the normals used to compute the winding number field. Since the WNNC property itself does not resolve the inside/outside orientation ambiguity, we further propose to incorporate an objective function from Parametric Gauss Reconstruction (PGR). We propose to iteratively update normals by alternating between WNNC-based normal updates and PGR-based gradient descents, which leads to an embarrassingly simple yet effective iterative algorithm that allows fast and high-quality convergence to a globally consistent normal vector field. Furthermore, our proposed algorithm only involves repeatedly evaluating the winding number formula and its derivatives, which can be accelerated and parallelized using a treecode-based approximation algorithm due to their special structures. Exploiting this fact, we implement a GPU-accelerated treecode-based solver. Our GPU (and even CPU) implementation can be significantly faster than the recent state-of-the-art methods for normal orientation from raw points. Our code is integrated with the popular PyTorch framework to facilitate further research into winding numbers, and is publicly available at https://jsnln.github.io/wnnc/index.html.

The non-line-of-sight imaging technique aims to reconstruct targets from multiply reflected light. For most existing methods, dense points on the relay surface are raster scanned to obtain high-quality reconstructions, which requires a long acquisition time. In this work, we propose a signal-surface collaborative regularization (SSCR) framework that provides noise-robust reconstructions with a minimal number of measurements. Using Bayesian inference, we design joint regularizations of the estimated signal, the$3D$voxel-based representation of the objects, and the 2D surface-based description of the targets. To our best knowledge, this is the first work that combines regularizations in mixed dimensions for hidden targets. Experiments on synthetic and experimental datasets illustrated the efficiency of the proposed method under both confocal and non-confocal settings. We report the reconstruction of the hidden targets with complex geometric structures with only 5 × 5 confocal measurements from public datasets, indicating an acceleration of the conventional measurement process by a factor of 10,000. Besides, the proposed method enjoys low time and memory complexity with sparse measurements. Our approach has great potential in real-time non-line-of-sight imaging applications such as rescue operations and autonomous driving.

Non-line-of-sight (NLOS) imaging aims at reconstructing the location, shape, albedo, and surface normal of the hidden object around the corner with measured transient data. Due to its strong potential in various fields, it has drawn much attention in recent years. However, long exposure time is not always available for applications such as auto-driving, which hinders the practical use of NLOS imaging. Although scanning fewer points can reduce the total measurement time, it also brings the problem of imaging quality degradation. This paper proposes a general learning-based pipeline for increasing imaging quality with only a few scanning points. We tailor a neural network to learn the operator that recovers a high spatial resolution signal. Experiments on synthetic and measured data indicate that the proposed method provides faithful reconstructions of the hidden scene under both confocal and non-confocal settings. Compared with original measurements, the acquisition of our approach is 16 times faster while maintaining similar reconstruction quality. Besides, the proposed pipeline can be applied directly to existing optical systems and imaging algorithms as a plug-in-and-play module. We believe the proposed pipeline is powerful in increasing the frame rate in NLOS video imaging.

We propose Parametric Gauss Reconstruction (PGR) for surface reconstruction from point clouds without normals. Our insight builds on the Gauss formula in potential theory, which represents the indicator function of a region as an integral over its boundary. By viewing surface normals and surface element areas as unknown parameters, the Gauss formula interprets the indicator as a member of some parametric function space. We can solve for the unknown parameters using the Gauss formula and simultaneously obtain the indicator function. Our method bypasses the need for accurate input normals as required by most existing non-data-driven methods, while also exhibiting superiority over data-driven methods, since no training is needed. Moreover, by modifying the Gauss formula and employing regularization, PGR also adapts to difficult cases such as noisy inputs, thin structures, sparse or nonuniform points, for which accurate normal estimation becomes quite difficult. Our code is publicly available at https://github.com/jsnln/ParametricGaussRecon .

Numerical Partial Differential Equation (PDE) solvers often require discretizing the physical domain by using a mesh. Mesh movement methods provide the capability to improve the accuracy of the numerical solution without introducing extra computational burden to the PDE solver, by increasing mesh resolution where the solution is not well-resolved, whilst reducing unnecessary resolution elsewhere. However, sophisticated mesh movement methods, such as the Monge-Ampère method, generally require the solution of auxiliary equations. These solutions can be extremely expensive to compute when the mesh needs to be adapted frequently. In this paper, we propose to the best of our knowledge the first learning-based end-to-end mesh movement framework for PDE solvers. Key requirements of learning-based mesh movement methods are: alleviating mesh tangling, boundary consistency, and generalization to mesh with different resolutions. To achieve these goals, we introduce the neural spline model and the graph attention network (GAT) into our models respectively. While the Neural-Spline based model provides more flexibility for large mesh deformation, the GAT based model can handle domains with more complicated shapes and is better at performing delicate local deformation. We validate our methods on stationary and time-dependent, linear and non-linear equations, as well as regularly and irregularly shaped domains. Compared to the traditional Monge-Ampère method, our approach can greatly accelerate the mesh adaptation process by three to four orders of magnitude, whilst achieving comparable numerical error reduction.

Low-dose computed tomography (LDCT) scans, which can effectively alleviate the radiation problem, will degrade the imaging quality. In this paper, we propose a novel LDCT reconstruction network that unrolls the iterative scheme and performs in both image and manifold spaces. Because patch manifolds of medical images have low-dimensional structures, we can build graphs from the manifolds. Then, we simultaneously leverage the spatial convolution to extract the local pixel-level features from the images and incorporate the graph convolution to analyze the nonlocal topological features in manifold space. The experiments show that our proposed method outperforms both the quantitative and qualitative aspects of state-of-the-art methods. In addition, aided by a projection loss component, our proposed method also demonstrates superior performance for semi-supervised learning. The network can remove most noise while maintaining the details of only 10% (40 slices) of the training data labeled.

Training process is crucial for the deployment of the network in applications which have two strict requirements on both accuracy and robustness. However, most existing approaches are in a dilemma, i.e. model accuracy and robustness form an embarrassing tradeoff - the improvement of one leads to the drop of the other. The challenge remains as for we try to improve the accuracy and robustness simultaneously. In this paper, we propose a novel training method via introducing the auxiliary classifiers for training on corrupted samples, while the clean samples are normally trained with the primary classifier. In the training stage, a novel distillation method named input-aware self distillation is proposed to facilitate the primary classifier to learn the robust information from auxiliary classifiers. Along with it, a new normalization method - selective batch normalization is proposed to prevent the model from the negative influence of corrupted images. At the end of training period, a L2-norm penalty is applied to the weights of primary and auxiliary classifiers such that their weights are asymptotically identical. In the stage of inference, only the primary classifier is used and thus no extra computation and storage are needed. Extensive experiments on CIFAR10, CIFAR100 and ImageNet show that noticeable improvements on both accuracy and robustness can be observed by the proposed auxiliary training. On average, auxiliary training achieves 2.21% accuracy and 21.64% robustness (measured by corruption error) improvements over traditional training methods on CIFAR100. Codes has been released on github.

Although ordinary differential equations (ODEs) provide insights for designing network architectures, its relationship with the non-residual convolutional neural networks (CNNs) is still unclear. In this paper, we present a novel ODE model by adding a damping term. It can be shown that the proposed model can recover both a ResNet and a CNN by adjusting an interpolation coefficient. Therefore, the damped ODE model provides a unified framework for the interpretation of residual and non-residual networks. The Lyapunov analysis reveals better stability of the proposed model, and thus yields robustness improvement of the learned networks. Experiments on a number of image classification benchmarks show that the proposed model substantially improves the accuracy of ResNet and ResNeXt over the perturbed inputs from both stochastic noise and adversarial attack methods. Moreover, the loss landscape analysis demonstrates the improved robustness of our method along the attack direction.

Feature distillation, a primary method in knowledge distillation, always leads to significant accuracy improvements. Most existing methods distill features in the teacher network through a manually designed transformation. In this paper, we propose a novel distillation method named task-oriented feature distillation (TOFD) where the transformation is convolutional layers that are trained in a data-driven manner by task loss. As a result, the task-oriented information in the features can be captured and distilled to students. Moreover, an orthogonal loss is applied to the feature resizing layer in TOFD to improve the performance of knowledge distillation. Experiments show that TOFD outperforms other distillation methods by a large margin on both image classification and 3D classification tasks. Codes have been released in Github.

Missing data recovery is an important and yet challenging problem in imaging and data science. Successful models often adopt certain carefully chosen regularization. Recently, the low dimensional manifold model (LDMM) was introduced by [S. Osher, Z. Shi, and W. Zhu, Low Dimensional Manifold Model for Image Processing, Technical report, cam report 16-04, UCLA, Los Angeles, CA, 2016] and shown to be effective in image inpainting. The authors of [ Low Dimensional Manifold Model for Image Processing, Technical report, cam report 16-04, UCLA, Los Angeles, CA, 2016] observed that enforcing low dimensionality on the image patch manifold serves as a good image regularizer. In this paper, we observe that having only the low dimensional manifold regularization is not enough sometimes, and we need smoothness as well. For that, we introduce a new regularization by combining the low dimensional manifold regularization with a higher order \bf CUrvature \bf REgularization, which we call new regularization CURE for short. The key step of CURE is to solve a biharmonic equation on a manifold. We further introduce a weighted version of CURE, called WeCURE, in a similar manner as the weighted nonlocal Laplacian (WNLL) method [Z. Shi, S. Osher, and W. Zhu, Weighted nonlocal Laplacian on interpolation from sparse data, J. Sci. Comput., 73 (2017), pp. 1164--1177]. Numerical experiments for image inpainting and semisupervised learning show that the proposed CURE and WeCURE significantly outperform LDMM and WNLL, respectively.

We unify the theory of optimal control of transport equations with the practice of training and testing of ResNets. Based on this unified viewpoint, we propose a simple yet effective ResNets ensemble algorithm to boost the accuracy of the robustly trained model on both clean and adversarial images. The proposed algorithm consists of two components: First, we modify the base ResNets by injecting a variance specified Gaussian noise to the output of each residual mapping. Second, we average over the production of multiple jointly trained modified ResNets to get the final prediction. These two steps give an approximation to the Feynman-Kac formula for representing the solution of a convection-diffusion equation. For the CIFAR10 benchmark, this simple algorithm leads to a robust model with a natural accuracy of {\bf 85.62}\% on clean images and a robust accuracy of ${\bf 57.94 \%}$ under the 20 iterations of the IFGSM attack, which outperforms the current state-of-the-art in defending against IFGSM attack on the CIFAR10.

The dipole orientation of fluorophores could be resolved by fluorescence polarization microscopy (FPM), which in turn reveals structural specificity for the labeled organelles. Conventional FPM can detect only the averaged fluorescence anisotropy collected from dipoles within the diffraction-limited volume. Super-resolution dipole orientation mapping (SDOM) method, which applies sparse deconvolution and least square estimation to fluorescence polarization modulation data, achieves the dipole orientation measurement within a sub-diffraction focal area. However, during SDOM analysis, some pixels with fluorescence signal are not resolved with orientation for relatively small adjusted R2. Here we report group-sparsity-based SDOM (GS-SDOM), which utilizes the relevance of modulation sequences to effectively improve the SDOM reconstruction model. More credible resolved dipole orientations with higher adjusted R2can be mapped and false positive estimation for local dipole orientation is vitally corrected. In addition to achieving the same spatial super-resolution as SDOM does, GS-SDOM accesses more morphological information with more credible orientations and more accurate local dipole distribution estimation. During the GS-SDOM analysis of actin filaments in mammalian kidney cells, the dipole orientation of fluorescence is detected always parallel to the direction of the actin filaments. Also with dipole orientation information extracted by GS-SDOM, the reconstructed visual circle from intensity dimension is discerned as jointed by double close filaments and 3-dimensional co-localization is accomplished in the intersection of actin filaments.

In this article, we introduce a surface reconstruction method that has excellent performance despite nonuniformly distributed, noisy, and sparse data. We reconstruct the surface by estimating an implicit function and then obtain a triangle mesh by extracting an iso-surface. Our implicit function takes advantage of both the indicator function and the signed distance function. The implicit function is dominated by the indicator function at the regions away from the surface and is approximated (up to scaling) by the signed distance function near the surface. On one hand, the implicit function is well defined over the entire space for the extracted iso-surface to remain near the underlying true surface. On the other hand, a smooth iso-surface can be extracted using the marching cubes algorithm with simple linear interpolations due to the properties of the signed distance function. Moreover, our implicit function can be estimated directly from an explicit integral formula without solving any linear system. An approach called disk integration is also incorporated to improve the accuracy of the implicit function. Our method can be parallelized with small overhead and shows compelling performance in a GPU version by implementing this direct and simple approach. We apply our method to synthetic and real-world scanned data to demonstrate the accuracy, noise resilience, and efficiency of this method. The performance of the proposed method is also compared with several state-of-the-art methods.

We replace the output layer of deep neural nets, typically the softmax function, by a novel interpolating function. And we propose end-to-end training and testing algorithms for this new architecture. Compared to classical neural nets with softmax function as output activation, the surrogate with interpolating function as output activation combines advantages of both deep and manifold learning. The new framework demonstrates the following major advantages: First, it is better applicable to the case with insufficient training data. Second, it significantly improves the generalization accuracy on a wide variety of networks. The algorithm is implemented in PyTorch, and the code is available at https://github.com/ BaoWangMath/DNN-DataDependentActivation.

In this paper, we propose a novel low dimensional manifold model (LDMM) and apply it to some image processing problems. LDMM is based on the fact that the patch manifolds of many natural images have low dimensional structure. Based on this fact, the dimension of the patch manifold is used as a regularization to recover the image. The key step in LDMM is to solve a Laplace--Beltrami equation over a point cloud which is solved by the point integral method. The point integral method enforces the sample point constraints correctly and gives better results than the standard graph Laplacian. Numerical simulations in image denoising, inpainting, and superresolution problems show that LDMM is a powerful method in image processing.