Dataset distillation aims to learn a small synthetic dataset that preserves most of the information from the original dataset. Dataset distillation can be formulated as a bi-level meta-learning problem where the outer loop optimizes the meta-dataset and the inner loop trains a model on the distilled data. Meta-gradient computation is one of the key challenges in this formulation, as differentiating through the inner loop learning procedure introduces significant computation and memory costs. In this paper, we address these challenges using neural Feature Regression with Pooling (FRePo), achieving the state-of-the-art performance with an order of magnitude less memory requirement and two orders of magnitude faster training than previous methods. The proposed algorithm is analogous to truncated backpropagation through time with a pool of models to alleviate various types of overfitting in dataset distillation. FRePo significantly outperforms the previous methods on CIFAR100, Tiny ImageNet, and ImageNet-1K. Furthermore, we show that high-quality distilled data can greatly improve various downstream applications, such as continual learning and membership inference defense. Please check out our webpage at https://sites.google.com/view/frepo.

Batch Normalization (BN) is a popular technique for training Deep Neural Networks (DNNs). BN uses scaling and shifting to normalize activations of mini-batches to accelerate convergence and improve generalization. The recently proposed Iterative Normalization (IterNorm) method improves these properties by whitening the activations iteratively using Newton’s method. However, since Newton’s method initializes the whitening matrix independently at each training step, no information is shared between consecutive steps. In this work, instead of exact computation of whitening matrix at each time step, we estimate it gradually during training in an online fashion, using our proposed Stochastic Whitening Batch Normalization (SWBN) algorithm. We show that while SWBN improves the convergence rate and generalization of DNNs, its computational overhead is less than that of IterNorm. Due to the high efficiency of the proposed method, it can be easily employed in most DNN architectures with a large number of layers. We provide comprehensive experiments and comparisons between BN, IterNorm, and SWBN layers to demonstrate the effectiveness of the proposed technique in conventional (many-shot) image classification and few-shot classification tasks.

For machine learning models to make reliable predictions in deployment, one needs to ensure the previously unknown test samples need to be sufficiently similar to the training data. The commonly used shift-invariant kernels do not have the compositionality and fail to capture invariances in high-dimensional data in computer vision. We propose a shift-invariant convolutional neural tangent kernel (SCNTK) based outlier detector and two-sample tests with maximum mean discrepancy (MMD) that is O(n) in the number of samples due to using the random feature approximation. On MNIST and CIFAR10 with various types of dataset shifts, we empirically show that statistical tests with such compositional kernels, inherited from infinitely wide neural networks, achieve higher detection accuracy than existing non-parametric methods. Our method also provides a competitive alternative to adapted kernel methods that require a training phase.

Deterministic down-sampling of an unordered point cloud in a deep neural network has not been rigorously studied so far. Existing methods down-sample the points regardless of their importance for the network output and often address down-sampling the raw point cloud before processing. As a result, some important points in the point cloud may be removed, while less valuable points may be passed to next layers. In contrast, the proposed adaptive down-sampling method samples the points by taking into account the importance of each point, which varies according to application, task and training data. In this paper, we propose a novel deterministic, adaptive, permutation-invariant down-sampling layer, called Critical Points Layer (CPL), which learns to reduce the number of points in an unordered point cloud while retaining the important (critical) ones. Unlike most graph-based point cloud down-sampling methods that use k-NN to find the neighboring points, CPL is a global down-sampling method, rendering it computationally very efficient. The proposed layer can be used along with a graph-based point cloud convolution layer to form a convolutional neural network, dubbed CP-Net in this paper. We introduce a CP-Net for 3D object classification that achieves high accuracy for the ModelNet 40 dataset among point cloud-based methods, which validates the effectiveness of the CPL.

We present a learning-based method to estimate the object bounding box from its 2D bird's-eye view (BEV) LiDAR points. Our method, entitled BoxNet, exploits a simple deep neural network that can efficiently handle unordered points. The method takes as input the 2D coordinates of all the points and the output is a vector consisting of both the box pose (position and orientation in LiDAR coordinate system) and its size (width and length). In order to deal with the angle discontinuity problem, we propose to estimate the double-angle sinusoidal values rather than the angle itself. We also predict the center relative to the point cloud mean to boost the performance of estimating the location of the box. The proposed method does not rely on the ordering of points as in many existing approaches, and can accurately predict the actual size of the bounding box based on the prior information that is obtained from the training data. BoxNet is validated using the KITTI 3D object dataset, with significant improvement compared with the state-of-the-art non-learning based methods.

Many successful image quality metrics rely on the structural information in an image to assess its perceptual quality. Extracting the structural information that is perceptually meaningful to our visual system, however, is a challenging task. This paper proposes a new quality assessment metric that relies on a sparse modeling approach to learn the inherent structures of the image. These structures are learnt as a set of basis vectors, such that any structure in the image can be represented by a linear combination of only a few of these basis vectors. This strategy is known to generate basis vectors that are qualitatively similar to the receptive field of the simple cells present in the mammalian primary visual cortex. The perceptual quality of the distorted image is estimated by comparing the structures of the reference and the distorted images in terms of the learnt basis vectors. Our approach is evaluated on five standard subject-rated image quality assessment datasets. The proposed metric exhibits high correlation with the subjective ratings outperforming several well established methods.

A wavelet-based image gradient transform is proposed. The proposed transform, called multi-scale gradient transform (MSGT), obtains the first order derivative of an image in terms of the wavelet detail coefficients. While traditional methods estimate the image gradients at each wavelet scale in terms of the horizontal and vertical wavelet coefficients only, the proposed transform obtains the gradients in terms of the diagonal, as well as the horizontal and vertical wavelet coefficients. The proposed MSGT is designed to be invertible, non-redundant and computationally efficient. We demonstrate the potential applications of the proposed transform in texture feature extraction, multi-scale edge detection, image quality assessment and image watermarking.

A robust data hiding scheme that embeds the watermark bits by quantizing the gradient directions of an image is proposed. By embedding the watermark in the angle, the watermark becomes robust to amplitude scaling attacks. To keep the watermark imperceptible and enhance its robustness, it is embedded in the significant gradient vectors of the image. The significant gradient vectors are obtained using discrete wavelet transform (DWT). Thus, the gradient vector at a pixel is first obtained in terms of the DWT coefficients. Then the gradient direction is quantized by modifying the DWT coefficients corresponding to the gradient vectors. Experimental results confirm that the proposed gradient direction watermarking (GDWM) method is robust to various types of attacks, specially amplitude scaling at tacks, and results in watermarked images of high fidelity.

We propose a robust quantization-based image watermarking scheme, called the gradient direction watermarking (GDWM), based on the uniform quantization of the direction of gradient vectors. In GDWM, the watermark bits are embedded by quantizing the angles of significant gradient vectors at multiple wavelet scales. The proposed scheme has the following advantages: 1) increased invisibility of the embedded watermark because the watermark is embedded in significant gradient vectors, 2) robustness to amplitude scaling attacks because the watermark is embedded in the angles of the gradient vectors, and 3) increased watermarking capacity as the scheme uses multiple-scale embedding. The gradient vector at a pixel is expressed in terms of the discrete wavelet transform (DWT) coefficients. To quantize the gradient direction, the DWT coefficients are modified based on the derived relationship between the changes in the coefficients and the change in the gradient direction. Experimental results show that the proposed GDWM outperforms other watermarking methods and is robust to a wide range of attacks, e.g., Gaussian filtering, amplitude scaling, median filtering, sharpening, JPEG compression, Gaussian noise, salt & pepper noise, and scaling.

Gradient estimators are mostly designed to yield accurate and robust estimates of the gradient magnitude, not the gradient direction. This paper proposes a method for the accurate and robust estimation of both the gradient magnitude and direction. It robustly estimates the gradient in the x- and y-directions. The robustness against noise is achieved by prefiltering and postfiltering of the gradient in each direction. To reduce edge blurring effects introduced by these filters, the gradient in a certain direction is obtained by applying the prefilter and postfilter in the perpendicular direction. The basic elements employed in each window are: highpass, lowpass and aggregation operators. The highpass operator is used as a gradient estimator, the lowpass operator is for prefiltering and postfiltering, and the aggregation operator is for aggregating the prefiltered and postfiltered gradients. Four different combinations of highpass, lowpass and aggregation operators are proposed: MVD-Median-Mean, MVD-Median-Max, RCMG-Median-Mean, and RCMG-Median-Max. Experimental results show that the RCMG-Median-Mean has the best performance in estimating the gradient and detecting the edges in noisy color images. It is computationally more efficient than the state-of-the-art gradient estimators and is able to accurately estimate the gradient direction as well as the gradient magnitude. Computer simulation results show that the proposed method outperforms other recently proposed color gradient estimators and edge detectors.

A new vector-wise scheme for the gradient estimation and edge detection in noisy color images is proposed. In color images, different types of noise may corrupt the image. To reduce the effects of noise in the gradient estimation, we introduce the RCMG-Median-Mean estimator. RCMG-Median-Mean is a combination of the robust color morphological gradient (RCMG), the median and the mean filters to accurately estimate both the gradient magnitude and the gradient direction at each pixel of a noisy color image. The simulation results show that the proposed method more accurately estimates the true gradient vector, has better corner detection and leads to better continuous edges with less computational complexity than the RCMG method.

This paper presents the new concept of watermark survival chance (WSC) for improving watermark robustness. WSC provides a robustness measure for an image feature (e.g. a discrete wavelet transform (DWT) coefficient) when used for watermark embedding, and thus can provide the watermark designer with prior knowledge on robust image features. As an illustrative example, we study additive spread spectrum watermarking in the DWT domain and consider JPEG compression as the attack. WSC is obtained for each DWT coefficient/subband for different compression ratios. Based on the WSC table for JPEG compression distortion, we suggest that: Wavelet coefficients can be divided into two main categories: block boundary coefficients and block non-boundary coefficients; block boundary coefficients generally are more robust for watermark embedding than block non-boundary coefficients; larger scale wavelet coefficients are generally more robust than smaller scales; a vertical subband is slightly preferred at small and large scales, while a horizontal subband is preferred at a medium scale.

A new vector-wise scheme for the gradient estimation in noisy color images is proposed. In the color images, different types of noise may corrupt the image. To reduce the effect of noise in the gradient estimation, the distance weighted order statistic filter (DWOSF) is introduced, to be embedded in the gradient estimator. DWOSF is an order statistic filter in which the color outliers in a neighborhood are adaptively assigned smaller weights than the other pixels in that neighborhood. The simulation results show that the proposed method accurately estimates the true gradient vector in noisy RGB color images.

An improved blind image quality assessment scheme that is based on the Watson's just noticeable difference (JND) modulated spread spectrum watermarking, is proposed. For the purpose of quality monitoring, a watermark is embedded into the original image. The image quality is estimated based on the detected watermark at the receiver side. In terms of the peak signal-to-noise ratio (PSNR), the proposed method is shown to be more robust and less perceptible than simple spread spectrum watermarking. This is due to several factors: an optimum detector is used for watermark detection, the watermark is appropriately selected from a set and the detector parameter is adjusted accordingly to closely yield an empirical ideal quality curve. The proposed method was tested by finding the quality estimates of different images compressed with different quality factors. The results indicate that the method can accurately estimate the quality of the received images based on the detected watermark power.