A recent trend in explainable AI research has focused on surrogate modeling, where neural networks are approximated as simpler ML algorithms such as kernel machines. A second trend has been to utilize kernel functions in various explain-by-example or data attribution tasks. In this work, we combine these two trends to analyze approximate empirical neural tangent kernels (eNTK) for data attribution. Approximation is critical for eNTK analysis due to the high computational cost to compute the eNTK. We define new approximate eNTK and perform novel analysis on how well the resulting kernel machine surrogate models correlate with the underlying neural network. We introduce two new random projection variants of approximate eNTK which allow users to tune the time and memory complexity of their calculation. We conclude that kernel machines using approximate neural tangent kernel as the kernel function are effective surrogate models, with the introduced trace NTK the most consistent performer.

We investigate the spectral properties of linear-width feed-forward neural networks, where the sample size is asymptotically proportional to network width. Empirically, we show that the spectra of weight in this high dimensional regime are invariant when trained by gradient descent for small constant learning rates; we provide a theoretical justification for this observation and prove the invariance of the bulk spectra for both conjugate and neural tangent kernels. We demonstrate similar characteristics when training with stochastic gradient descent with small learning rates. When the learning rate is large, we exhibit the emergence of an outlier whose corresponding eigenvector is aligned with the training data structure. We also show that after adaptive gradient training, where a lower test error and feature learning emerge, both weight and kernel matrices exhibit heavy tail behavior. Simple examples are provided to explain when heavy tails can have better generalizations. We exhibit different spectral properties such as invariant bulk, spike, and heavy-tailed distribution from a two-layer neural network using different training strategies, and then correlate them to the feature learning. Analogous phenomena also appear when we train conventional neural networks with real-world data. We conclude that monitoring the evolution of the spectra during training is an essential step toward understanding the training dynamics and feature learning.

Data mining is a technique to discover patterns and trends in data and can be used to create a model to predict those patterns and trends. This is particularly useful for data sets that are not amenable to traditional statistical analysis. One particular data mining task is classification, predicting a quantity that can only take on a finite number of values. An important class of binary classifiers are support vector machines (SVMs). Traditional SVMs use constrained optimization to find a separating hyperplane. A new data point is classified based on which side of the separating hyperplane it happens to fall on. All SVMs try to minimize the number of potential errors the classifier makes by minimizing a sum of distances from the hyperplane. However, the actual task of classification does not place any importance on a distance. In order to model this more closely, we propose the integer support vector machine classifier (ISVM). ISVM uses binary indicator error variables to directly minimize the number of potential errors the classifier can make.

The oligopoly model of international fishing of Szidarovszky and Okuguchi (1998) is revisited with the additional assumption that there is a time lag in obtaining and implementing information on the fish stock. The local asymptotical behavior of the equilibrium is analyzed including the birth of limit cycles.