Estimation of a regression function from independent and identically distributed random variables is considered. The$L_{2}$error with integration with respect to the design measure is used as an error criterion. Over-parametrized deep neural network estimates are defined which are based on a special network topology, which use a special random initialization and where all the weights are learned by the gradient descent. It is shown that the expected$L_{2}$error of these estimates converges to zero with the rate close to$n^{-1/(1+d)}$in case that the regression function is Hölder smooth with Hölder exponent$p \in [{1/2,1}]$. In case of an interaction model where the regression function is assumed to be a sum of Hölder smooth functions where each of the functions depends only on$d^{*}$of ofdcomponents of the design variable, it is shown that these estimates achieve the corresponding$d^{*}$-dimensional rate of convergence.

Estimation of univariate regression function by a neural network with one hidden layer is considered, where the weight vector is determined by applying gradient descent to a regularized empirical$L_{2}$risk. Here the number of hidden neurons is allowed to be much larger than the sample size. It is shown that the estimate nevertheless generalizes well in case that the Fourier transform of the regression function decays suitably fast, and that in this case over-parametrization leads to a particular good rate of convergence.

Image classifiers based on over-parametrized deep convolutional neural networks with an average-pooling are proposed. The weights of the network are learned by gradient descent. We present the bound on the rate of convergence of the difference between the expected misclassification risk of the plug-in classifier and the Bayes risk. The obtained rate of convergence is independent of image dimension under appropriate constraints on the image distribution.

Abstract Dynamic latent scale GAN is an architecture‐agnostic encoder‐based generative model inversion method. This paper introduces a method to efficiently integrate perceptual VAE into dynamic latent scale GAN to improve the performance of dynamic latent scale GAN. When dynamic latent scale GAN is trained with a normal i.i.d. latent random variable and the latent encoder is integrated into the discriminator, a sum of a predicted latent random variable of real data and a scaled normal noise follows the normal i.i.d. random variable. Since this random variable is paired with real data and follows the latent random variable, it can be used for both VAE and GAN training. Furthermore, by considering the intermediate layer output of the discriminator as the feature encoder output, the VAE can be trained to minimise the perceptual reconstruction loss. The forward propagation & backpropagation for minimising this perceptual reconstruction loss can be integrated with those of GAN training. Therefore, the proposed method does not require additional computations compared to typical GAN or dynamic latent scale GAN. Integrating perceptual VAE to dynamic latent scale GAN improved the generative and inversion performance of the model.

Various methods for fitting an unknown functions from the set of noisy measurements are applicable to a wide variety of problems. Among them, the nonparametric algorithms based on the Parzen kernel are willingly used. In the article, we propose a novel and very effective numerical simplification in Parzen approach leading to a significant reduction in computation time. The algorithm is basically developed for multidimensional case. The two-dimensional version of the method is explained in details and analysed. Computational complexity and speed of convergence of the algorithm are studied. Some applications for solving real problems with our algorithms are presented.

This paper considers different deep learning architectures to learn patterns that are objects lying on the Riemannian and Grassmann manifolds. Among them, we considered cascades of classifier ensembles (CCEs), convolutional neural networks (CNNs), and deep neural forests (DNFs). All aforementioned architectures have linearized and nonlinearized versions. Patterns that are objects of Riemannian manifolds are classifier prediction pairwise matrices (CPPMs) while objects of the Grassmann manifolds are obtained using decision profiles (DPs). We also compared our architectures with CCEs that operate in the Euclidean geometry. As seen from the experimental results deep learning architectures based on CNNs provided the best results.

Deep neural networks (DNNs) achieve impressive results for complicated tasks like object detection on images and speech recognition. Motivated by this practical success, there is now a strong interest in showing good theoretical properties of DNNs. To describe for which tasks DNNs perform well and when they fail, it is a key challenge to understand their performance. The aim of this paper is to contribute to the current statistical theory of DNNs. We apply DNNs on high dimensional data and we show that the least squares regression estimates using DNNs are able to achieve dimensionality reduction in case that the regression function has locally low dimensionality. Consequently, the rate of convergence of the estimate does not depend on its input dimension$d$, but on its local dimension$d^{*}$and the DNNs are able to circumvent the curse of dimensionality in case that$d^{*}$is much smaller than$d$. In our simulation study we provide numerical experiments to support our theoretical result and we compare our estimate with other conventional nonparametric regression estimates. The performance of our estimates is also validated in experiments with real data.

In this article we consider asymptotic properties of the normalized radial basis function networks with one hidden layer trained by independent patterns with arbitrary distributions. Convergence and rates of convergence are investigated and the choice of the radial basis functions and the network parameters are discussed.

This paper introduces a new topic and research of geometric classifier ensemble learning using two types of objects: classifier prediction pairwise matrix (CPPM) and decision profiles (DPs). Learning from CPPM requires using Riemannian manifolds (R-manifolds) of symmetric positive definite (SPD) matrices. DPs can be used to build a Grassmann manifold (G-manifold). Experimental results show that classifier ensembles and their cascades built using R-manifolds are less dependent on some properties of individual classifiers (e.g. depth of decision trees in random forests (RFs) or extra trees (ETs)) in comparison to G-manifolds and Euclidean geometry. More independent individual classifiers allow obtaining R-manifolds with better properties for classification. Generally, the accuracy of classification in nonlinear geometry is higher than in Euclidean one. For multi-class problems, G-manifolds perform similarly to stacking-based classifiers built on R-manifolds of SPD matrices in terms of classification accuracy.

This paper considers three different classifier stacking algorithms: simple stacking, cascades of classifier ensembles and nonlinear version of classifier stacking based on classifier interactions. Classifier interactions can be expressed using classifier prediction pairwise matrix (CPPM). As a meta-learner for the last algorithm Convolutional Neural Networks (CNNs) and two other classifier stacking algorithms (simple classifier stacking and cascades of classifier ensembles) have been applied. This allows applying classical stacking and cascade-based recursive stacking in the Euclidean and the Riemannian geometries. The cascades of random forests (RFs) and extra trees (ETs) are considered as a forest-based alternative to deep neural networks [1]. Our goal is to compare accuracies of the cascades of RFs and CNN-based stacking or deep multi-layer perceptrons (MLPs) for different classifications problems. We use gesture phase dataset from UCI repository [2] to compare and analyze cascades of RFs and extra trees (ETs) in both geometries and CNN-based version of classifier stacking. This data set was selected because generally motion is considered as a nonlinear process (patterns do no lie in Euclidean vector space) in computer vision applications. Thus we can assess how good are forest-based deep learning and the Riemannian manifolds (R-manifolds) when applied to nonlinear processes. Some more datasets from UCI repository were used to compare the aforementioned algorithms to some other well-known classifiers and their stacking-based versions in both geometries. Experimental results show that classifier stacking algorithms in Riemannian geometry (R-geometry) are less dependent on some properties of individual classifiers (e.g. depth of decision trees in RFs or ETs) in comparison to Euclidean geometry. More independent individual classifiers allow to obtain R-manifolds with better properties for classification. Generally, accuracy of classification using classifier stacking in R-geometry is higher than in Euclidean one.

We introduce and investigate in multiclass setting an efficient classifier which partitions the training data by means of multidimensional parallelepipeds called boxes. We show that multiclass classification problem at hand can be solved by combining the heuristic minimum clique cover approach and the k-nearest neighbor rule. Our algorithm is motivated by an algorithm for partitioning a graph into a minimal number of cliques. The main advantage of a new classifier called Box classifier is that it optimally utilizes the geometrical structure of the training set by reducing the 1-class classification problem to a single nearest neighbor problem. We discuss computational complexity of the proposed Box classifier. The extensive experiments performed on the simulated and real data from UCI Machine Learning Repository for binary and multiclass problems show that in almost all cases the Box classifier performs significantly better than k-NN, SVM and decision trees.

Uncertainty quantification of a technical system can be done using density estimation. We usually start with a stochastic model, which is fitted to the technical system, and the density estimation is done using data from this stochastic model. However, in any application, such a stochastic model will not be perfect, and the estimation of the density should take into account the inadequacy of the stochastic model. In this paper, we show how observed data of the real system together with an imperfect simulation model can be used to derive confidence bands for the density of the technical system. Our main result is that the newly introduced confidence bands allow to derive lower and upper bounds on the probability of intervals in the technical system. Furthermore, we present an upper bound on the area of the confidence band in case of a smooth density. The results are illustrated by applying the estimates to simulated and real data.

This paper is concerned with learning from predictions. Predictions are obtained by ensemble of classifiers such as random forests (RF) or extra-trees. One assumes that estimators are semi independent so that they can be considered as prediction space. Hence we project our feature vector to the space of estimators obtaining responses from each of them. The responses for RFs are conditional class probabilities. The responses might be considered as projections onto some direction in quasi-orthogonal space which are decision trees of a RF. After that one creates the connected Riemannian manifold by computing a matrix of pairwise products of predictions for all trees in the RF. These matrices are symmetric and positive definite which is a necessary and sufficient condition to have a connected Riemannian manifold (R manifold). Because outputs of trees are conditional probabilities we have to create as many such matrices as there are classes. Stacking all these matrices together we obtain a tensor which is passed to Convolutional Neural Networks (CNN) for learning. We tested our algorithm on 11 datasets from UCI repository representing difficult classification problems. The results show very fast learning and convergence of loss and prediction accuracy. The proposed algorithm outperforms feature-based classical classifier ensembles (RFs and extra-trees) for every tested dataset from UCI repository.

Let X be an Rdvalued random variable, let m : Rd→ R be a measurable function and set Y = m(X). Given a sample of (X, Y) of size n, we consider the problem of estimating the quantile of Y of a given level α ∈ (0, 1). A method for choosing the parameter of a surrogate model of m is introduced, and it is shown that the corresponding surrogate quantile estimate achieves the rate of convergence bounded by the sum of the minimal rate of convergence of the quantile estimates corresponding to the given surrogate estimates and a term of order log(n)/n. The finite sample size behavior of this quantile estimate is illustrated by applying it to simulated data and to a quantile estimation problem in mechanical engineering.

A prime factor deciding the survival rate of a breast cancer patient is the accuracy with which the malignancy grade of a breast tumor is determined. A Fine Needle Aspiration (FNA) biopsy is a key mechanism for breast cancer diagnosis as well as for assigning grades to malignant cases. In this paper, based on published cytological malignancy grading systems, we propose six computer-aided grading frameworks to assign malignancy grades to cytological images of FNA biopsies of breast cancer. The proposed computer-aided grading frameworks were tested on 332 FNA biopsy images composed of 66 images with high malignancy (G3) and 266 images with intermediate malignancy (G2) that were histopathologically validated using the Bloom-Richardson grading system. The best results were obtained for the Support Vector Machine classifier for computer-aided versions of the Robinson's and Khan et al.'s cytological grading systems with accuracies of 97.57% and 96.98% for case classification (where a case is a pair of 100× and 400× magnification images for a patient) and 95.23% and 98.36% for patient classification, respectively.

In this paper, we introduce the so-called hierarchical interaction models, where we assume that the computation of the value of a function m : ℝd→ ℝ is done in several layers, where in each layer a function of at most d* inputs computed by the previous layer is evaluated. We investigate two different regression estimates based on polynomial splines and on neural networks, and show that if the regression function satisfies a hierarchical interaction model and all occurring functions in the model are smooth, the rate of convergence of these estimates depends on d* (and not on d). Hence, in this case, the estimates can achieve good rate of convergence even for large d, and are in this sense able to circumvent the so-called curse of dimensionality.

This paper explores the supervised pattern recognition problem based on feature partitioning. This formulation leads to a new problem in computational geometry. The supervised pattern recognition problem is formulated as an heuristic good clique cover problem satisfying the k-nearest neighbors rule. First it is applied a heuristic algorithm for partitioning a graph into a minimal number of cliques. Next cliques are merged using the k-nearest neighbors rule. An important advantage of this approach is the decomposition of a problem involving l classes into l optimization problems involving a single class. The computational complexity of the method, computational procedures, and classification rules are discussed. A geometrical interpretation of the solution is also given. Using the proposed approach, the geometrical structure of the training set is utilized in the best possible way.

Regression estimation often suffers from the curse of dimensionality. In the present paper we circumvent this problem by introducing a class of models called hierarchical interaction models where the values of a function m ∶ ℝd→ ℝ are computed in a feed-forward manner in several layers, where in each layer a function of at most d* inputs produced by the previous layer is computed. We introduce regression estimates based on neural networks with two hidden layers and apply them to estimation of regression functions from a class of hierarchical interaction models. Under smoothness condition imposed on all functions occurring in the model we show that the rate of convergence of these estimates depends on d*, which is typically much smaller than d.

Summary Support vector machine (SVM) is one of the most popular classification techniques in pattern recognition community. However, because of outliers in the training samples, SVM tends to perform poorly under such circumstances. In this paper, we borrow the idea from compressive sensing by introducing an extra term to the objective function of the standard SVM in order to achieve a sparse representation. Furthermore, instead of using thel0norm, we adopt thel1norm in our sparse SVM. In most cases, our method achieves higher classification rates than the standard SVM because of sparser support vectors and is more robust to outliers in the datasets. Experimental results show that our proposed SVM is efficient in pattern recognition applications. Copyright © 2015 John Wiley & Sons, Ltd.

Summary We present a new method for automatic detection of suspicious breast cancer lesions using ultrasound. The system is fully automated. It uses fuzzy logic and compounding for de‐noising. A fuzzy membership function based on the gray values of ultrasound images is applied for de‐noising, improving the quality of the image and increasing separation between foreground and background, thus making easier detection of lesions. A novel approach based on neural network is used for segmentation of ultrasound images, and correlation between ultrasound images taken from different angles allows overcoming the problem of shadowing. We consider a combination of morphological and texture features and use sequential forward search, sequential backward search, and distance‐based method to select the best subset of features. We rank the features using distance‐based method and use a combination of sequential forward search and sequential backward search to select the best features (bidirectional search). Finally, support vector machine classifier is used for detecting suspicious lesions. The results of experiments show that our system performs better than other state‐of‐the‐art computer‐aided diagnosis systems with the accuracy of 98.75%. Furthermore, we used concurrency to improve the computational efficiency. In concurrent implementation of de‐noising, segmentation, and feature selection and extraction, we assign each pixel of an ultrasound image to a different thread. We also benefit from multi‐core computing by running each classifier on a different thread. Concurrent implementation of our computer‐aided diagnosis system reduces overall computational time by 85%. Copyright © 2015 John Wiley & Sons, Ltd.

Nonparametric estimation of a quantile qm(X),αof a random variable m(X) is considered, where m : ℝd→ ℝ is a function, which is costly to compute and X is an ℝd-valued random variable with known distribution. Monte Carlo surrogate quantile estimates are considered, where in a first step, the function m is estimated by some estimate (surrogate) mn, and then, the quantile qm(X),αis estimated by a Monte Carlo estimate of the quantile qmn(X),α. A general error bound on the error of this quantile estimate is derived, which depends on the local error of the function estimate mn, and the rates of convergence of the corresponding Monte Carlo surrogate quantile estimates are analyzed for two different function estimates. The finite sample size behavior of the estimates is investigated in simulations.

In this paper, we study the problem of density estimation from data that contains small measurement errors. The only assumption on these errors is that the maximal measurement error is bounded by some real number converging to zero for sample size tending to infinity. In particular, we do not assume that the measurement errors are independent with expectation zero. We estimate the density by a standard kernel density estimate applied to data with measurement errors and derive a data-driven method to choose its bandwidth. We derive an adaptation result for this estimate and analyze the expected L1error of our density estimate depending on the smoothness of the density and the size of the maximal measurement error. The results are applied in a density estimation problem in a simulation model, where we show under suitable assumptions that the L1error of our newly proposed estimate converges to zero much faster than the L1error of the standard kernel density estimate if both are based on the same number of observations in the simulation model. The performance of the method in case of finite sample size is analyzed using simulated data.

Let X, X1, X2, ... be independent and identically distributed ℝd-valued random variables and let m : ℝd→ ℝ be an unknown measurable function such that a density f of Y = m(X) exists. In this paper we consider estimating f based on i.i.d. sample (X1, Y1);...; (Xn, Yn) of (X, Y) and on additional independent observations of X. We compare the standard kernel density estimate based on the y-values of the sample of (X, Y) and a kernel density estimate based on artificially generated y-values corresponding to the additional observations of X. It is shown that under suitable smoothness assumptions on f and m the rate of convergence of the L1error of the latter estimate is better than that of the standard kernel density estimate. Furthermore, a density estimate defined as convex combination of these two estimates is considered and a data-driven choice of the bandwidths and the weight of the convex combination is proposed and investigated.

The effectiveness of the treatment of breast cancer depends on its timely detection. An early step in the diagnosis is the cytological examination of breast material obtained directly from the tumor. This work reports on advances in computer-aided breast cancer diagnosis based on the analysis of cytological images of fine needle biopsies to characterize these biopsies as either benign or malignant. Instead of relying on the accurate segmentation of cell nuclei, the nuclei are estimated by circles using the circular Hough transform. The resulting circles are then filtered to keep only high-quality estimations for further analysis by a support vector machine which classifies detected circles as correct or incorrect on the basis of texture features and the percentage of nuclei pixels according to a nuclei mask obtained using Otsu's thresholding method. A set of 25 features of the nuclei is used in the classification of the biopsies by four different classifiers. The complete diagnostic procedure was tested on 737 microscopic images of fine needle biopsies obtained from patients and achieved 98.51% effectiveness. The results presented in this paper demonstrate that a computerized medical diagnosis system based on our method would be effective, providing valuable, accurate diagnostic information.

We propose a new k-NN regression estimate based on a data-dependent metric in Rdwhich is used to define the k-nearest neighbors of a given point. The metric is invariant under all affine transformations. With this metric, the standard k-nearest neighbor regression estimate is asymptotically consistent under the usual conditions on k, and minimal requirements on the input data.

We consider nonlinear function estimation using Radial Basis Function Networks. We analytically determine the optimal radial kernel minimizing the Mean Integrated Square Error (MISE) and the optimal MISE rate of convergence. The rates of convergence for various classes of nonlinear functions and input densities are also considered.

The denoising of a natural signal/image corrupted by Gaussian white noise is a classical problem in signal/image processing. However, it is still in its infancy to denoise high dimensional data. In this paper, we extended Sendur and Selesnick's bivariate wavelet thresholding from two-dimensional (2D) image denoising to three-dimensional (3D) data cube denoising. Our study shows that bivariate wavelet thresholding is still valid for 3D data cubes. Experimental results show that bivariate wavelet thresholding on 3D data cube is better than performing 2D bivariate wavelet thresholding on every spectral band separately, VisuShrink, and Chen and Zhu's 3-scale denoising.

We consider American options also called Bermudan options in discrete time.We use the dual approach to derive upper bounds on the price of such options using only a reduced number of nested Monte Carlo steps. The key idea is to use nonparametric regression to estimate continuation values and all other required conditional expectations and to combine the resulting estimate with another estimate computed by using only a reduced number of nested Monte Carlo steps. The mean value of the resulting estimate is an upper bound on the option price. One can show that the estimates of the option prices are universally consistent, i.e., they converge to the true price regardless of the structure of the continuation values. The finite sample behavior is validated by experiments on simulated data.

A new courtesy amount recognition module of CENPARMIpsilas check reading system (CRS) is proposed in this paper. The module consists of 3 main segments: pre-processing, segmentation and recognition, and post-processing. A new feedback-based segmentation algorithm is adopted for the segmentation task. Besides one individual numeral recognizer for numerals from dasia0psila to dasia9psila, one convolutional neural network(CNN) recognizer for ldquo00rdquo and ldquo000rdquo numeral strings is also integrated into our module for the recognition task. The experimental results on the Quebec Bell Check database show that the recognition rate of the courtesy amount has improved from 41.2% to 74.3%.

When linear support vector machines (SVMs) are applied to multi-class text categorization in industry, the size of the linear SVM model is very large, usually greater than several gigabytes. As a result, the model cannot directly fit into the computer memory and the classification process is slow. In this paper, a novel method based on vector norm is proposed to shrink the model size significantly without sacrificing the classification accuracy. Also, we propose a cache-efficient implementation of multi-class linear SVMs in the classification phase. Our experimental results have shown that on Yahoo-Korea dataset the proposed method can shrink the model size from 5.2 gigabytes to 260 megabytes and the efficient implementation of linear SVM has obtained a speedup factor of 44.

This paper presents a multimodal approach for a biometrics verification system. It is based on face and hand images captured by a cell phone. The algorithm includes all parts that are required for face and hand verification, such as feature extraction, classification and authentication. To find local facial features, such as eyes, mouth and nose, we apply a point distribution model and active shape models. We use the same system to find distinctive points in hand geometry. The face feature vector is constructed by applying a Gabor filter to the image and extracting the key points found by an active shape model. The palm feature vector contains characteristics of the hand geometry features. A support vector machine (SVM) is applied to verify the identity of the user. One SVM machine is built for each person in the database to distinguish that person from others. To test the algorithm we built our own database containing face and hand images taken by a cell phone camera. The database contains 480 frontal face images and 120 hand images of 30 persons (16 face images and 4 hand images per person).

The rates of convergence of plug-in kernel, partitioning, and nearest neighbors classification rules are analyzed. A margin condition, which measures how quickly thea posterioriprobabilities cross the decision boundary, smoothness conditions on thea posterioriprobabilities, and boundedness of the feature vector are imposed. The rates of convergence of the plug-in classifiers shown in this paper are faster than previously known.

A new palmprint classification method is proposed in this paper by using the dual-tree complex wavelet transform. The dual-tree complex wavelet transform has such important properties as the approximate shift-invariance and high directional selectivity. These properties are very important in invariant palmprint classification. Support vector machines are used as a classifier and the Gaussian radial basis function kernel is selected in the experiments. Experimental results show that the dual-tree complex wavelet features outperform the scalar wavelet features, and three previously developed methods. We conclude that the dual-tree complex wavelet features should be used for invariant palmprint classification instead of the scalar wavelet features.

We discuss rates of convergence of plug-in kernel, partitioning and nearest neighbors classification rules under margin condition. Margin condition characterizes the rate with which a posteriori probabilities cross the decision boundary. We show the rates of convergence of the plug-in classifiers under smoothness conditions on a posteriori probabilities and assuming that feature vectors are contained in a compact set. We obtain particularly fast rates of convergence assuming, in addition, that feature vectors distributions have densities bounded away from zero.

This paper presents two fast and robust algorithms for word skew and slant corrections based on Radon transform. For the skew correction, we maximize a global measure which is defined by Radon transform of image and its gradient to estimate the slope. For the slant correction, Radon transform is used to estimate the long strokes and a word slant is measured by the average angle of these long strokes. Compared with the previous methods, these two algorithms do not require the setting of parameters heuristically. Moreover, the algorithms perform well on words of short length, where the traditional methods usually fail.

A fast iteration algorithm is proposed to approximate the reduced set vectors shared by each binary SVM solution for multi-class classification simultaneously. The iteration algorithm can be applied to the general kernel types such as k(/spl par/ x - x' /spl par//sup 2/) and k(x/sup T/x'). In addition, we present a fast block algorithm in the test phase to speed up the classification further. Experimental results have shown that the classification speeds on MNIST and Hanwang handwritten digit databases on P4 1.7 Ghz were about 16,000 and 10,895 patterns per second without sacrificing the classification accuracy of the original SVM system. The speed-up factor of 110 on MNIST database has been achieved.

We present a general bound on the expected L2error of adaptive least squares estimates. By applying it to multiple hidden layer feedforward neural network regression function estimates we are able to obtain optimal (up to log factor) rates of convergence for Lipschitz classes and fast rates of convergence for some classes of regression functions such as additive functions

Training a support vector machine on a data set of huge size with thousands of classes is a challenging problem. This paper proposes an efficient algorithm to solve this problem. The key idea is to introduce a parallel optimization step to quickly remove most of the nonsupport vectors, where block diagonal matrices are used to approximate the original kernel matrix so that the original problem can be split into hundreds of subproblems which can be solved more efficiently. In addition, some effective strategies such as kernel caching and efficient computation of kernel matrix are integrated to speed up the training process. Our analysis of the proposed algorithm shows that its time complexity grows linearly with the number of classes and size of the data set. In the experiments, many appealing properties of the proposed algorithm have been investigated and the results show that the proposed algorithm has a much better scaling capability than Libsvm, SVMlight, and SVMTorch. Moreover, the good generalization performances on several large databases have also been achieved.

This paper establishes weak and strong universal consistency of regression estimates based on normalized radial basis function networks when the network parameters are chosen by empirical risk minimization.

The denoising of a natural image corrupted by Gaussian noise is a classical problem in signal or image processing. Donoho and his coworkers at Stanford pioneered a wavelet denoising scheme by thresholding the wavelet coefficients arising from the standard discrete wavelet transform. This work has been widely used in science and engineering applications. However, this denoising scheme tends to kill too many wavelet coefficients that might contain useful image information. In this paper, we propose one wavelet image thresholding scheme by incorporating neighbouring coefficients, namely NeighShrink. This approach is valid because a large wavelet coefficient will probably have large wavelet coefficients as its neighbours. Experimental results show that NeighShrink is better than the Wiener filter and the conventional wavelet denoising approaches: VisuShrink and SUREShrink. We also investigate different neighbourhood sizes and find that a size of 3/spl times/3 is the best among all window sizes.

We prove a general bound on the expected L/sub 2/ error of adaptive least squares estimates. By applying it to multilayer feedforward neural network regression function estimates we are able to obtain fast rates of convergence in special classes of regression functions such as additive functions.

A fast support vector machine (SVM) training algorithm is proposed under SVM's decomposition framework by effectively integrating kernel caching, digest and shrinking policies and stopping conditions. Kernel caching plays a key role in reducing the number of kernel evaluations by maximal reusage of cached kernel elements. Extensive experiments have been conducted on a large handwritten digit database MNIST to show that the proposed algorithm is much faster than Keerthi et al.'s improved SMO, about nine times. Combined with principal component analysis, the total training for ten one-against-the-rest classifiers on MNIST took less than an hour. Moreover, the proposed fast algorithm speeds up SVM training without sacrificing the generalization performance. The 0.6% error rate on MNIST test set has been achieved. The promising scalability of the proposed scheme paves a new way to solve more large-scale learning problems in other domains such as data mining.

We consider the extension of the Whittaker-Shannon (WS) reconstruction formula to the case of signals sampled in the presence of noise and which are not necessarily band limited. Observing that in this situation the classical sampling expansion yields inconsistent reconstruction, we introduce a class of signal recovery methods with a smooth correction of the interpolation series. Two alternative data smoothing methods are examined based either on a global postfiltering or a local data presmoothing. We assess the accuracy of the methods by the global L/sub 2/ error. Both band-limited and non-band-limited signals are considered. A general class of correlated noise processes is taken into account. The weak and strong rates of convergence of the algorithms are established and their relative efficiency is discussed. The influence of noise memory and its moment structure on the accuracy is thoroughly examined.

Proposes an algorithm to find piecewise linear skeletons of handwritten characters by using principal curves. The development of the method was inspired by the apparent similarity between the definition of principal curves (smooth curves which pass through the "middle" of a cloud of points) and medial axes (smooth curves that run equidistantly from the contours of a character image). The central fitting-and-smoothing step of the algorithm is an extension of the polygonal line algorithm, which approximates principal curves of data sets by piecewise linear curves. The polygonal line algorithm is extended to find principal graphs and complemented with two steps specific to the task of skeletonization: an initialization method to capture the approximate topology of the character, and a collection of restructuring operations to improve the structural quality of the skeleton produced by the initialization method. An advantage of our approach over existing methods is that we optimize the skeleton graph by minimizing an intuitive and explicit objective function that captures the two competing criteria of smoothing the skeleton and fitting it closely to the pixels of the character image. We tested the algorithm on isolated handwritten digits and images of continuous handwriting. The results indicated that the proposed algorithm can find a smooth medial axis in the great majority of a wide variety of character templates and that it substantially improves the pixel-wise skeleton obtained by traditional thinning methods.

This paper proposes a general local learning framework to effectively alleviate the complexities of classifier design by means of "divide and conquer" principle and ensemble method. The learning framework consists of quantization layer and ensemble layer. After GLVQ and MLP are applied to the framework, the proposed method is tested on MNIST handwritten digit database. The obtained performance is very promising, an error rate with 0.99%, which is comparable to that of LeNet5, one of the best classifiers on this database. Further, in contrast to LeNet5, our method is especially suitable for a large-scale real-world classification problem.

We present multivariate penalized least squares regression estimates. We use Vapnik-Chervonenkis (see Statistical Learning Theory 1998) theory and bounds on the covering numbers to analyze convergence of the estimates. We show strong consistency of the truncated versions of the estimates without any conditions on the underlying distribution.

We propose an algorithm to find piecewise linear skeletons of handwritten characters by using principal curves. The development of the method was inspired by the apparent similarity between the definitions of principal curves (smooth curves which pass through the "middle" of a cloud of points) and the medial axis (smooth curves that go equidistantly from the contours of a character image). The algorithm is an extension of the polygonal line algorithm, originally designed to find principal curves of data sets, to compute the principal graph of a data set. Test results indicate that the proposed algorithm substantially improves the pixelwise skeleton obtained by traditional thinning methods.

We apply complexity regularization to learn normalized radial basis function networks in nonparametric classification. We study convergence and the rates of convergence of the empirically trained networks and verify the results in computer experiments.

Principal curves have been defined as "self-consistent" smooth curves which pass through the "middle" of a d-dimensional probability distribution or data cloud. They give a summary of the data and also serve as an efficient feature extraction tool. We take a new approach by defining principal curves as continuous curves of a given length which minimize the expected squared distance between the curve and points of the space randomly chosen according to a given distribution. The new definition makes it possible to theoretically analyze principal curve learning from training data and it also leads to a new practical construction. Our theoretical learning scheme chooses a curve from a class of polygonal lines with k segments and with a given total length to minimize the average squared distance over n training points drawn independently. Convergence properties of this learning scheme are analyzed and a practical version of this theoretical algorithm is implemented. In each iteration of the algorithm, a new vertex is added to the polygonal line and the positions of the vertices are updated so that they minimize a penalized squared distance criterion. Simulation results demonstrate that the new algorithm compares favorably with previous methods, both in terms of performance and computational complexity, and is more robust to varying data models.

In this paper we apply normalized radial basis function networks to function learning and in nonparametric classification. A simple parameter learning technique is proposed and convergence and the rates of convergence of the empirically trained networks are studied theoretically and in computer experiments.

In this paper we apply the method of complexity regularization to derive estimation bounds for nonlinear function estimation using a single hidden layer radial basis function network. Our approach differs from previous complexity regularization neural-network function learning schemes in that we operate with random covering numbers and l(1) metric entropy, making it possible to consider much broader families of activation functions, namely functions of bounded variation. Some constraints previously imposed on the network parameters are also eliminated this way. The network is trained by means of complexity regularization involving empirical risk minimization. Bounds on the expected risk in terms of the sample size are obtained for a large class of loss functions. Rates of convergence to the optimal loss are also derived.

Recursive radial basis function (RRBF) neural networks are introduced and discussed. We study in detail the nets with diagonal receptive field matrices. Parameters of the networks are learned by a simple procedure. Convergence and the rates of convergence of RRBF nets in the mean integrated absolute error (MIAE) sense are studied under mild conditions imposed on some of the network parameters. The obtained results also give the upper bounds on the performance of RRBF nets learned by minimizing the empirical L/sub 1/ error.

The method of complexity regularization is applied to one hidden-layer radial basis function networks to derive regression estimation bounds and convergence rates for classification. Bounds on the expected risk in terms of the training sample size are obtained for a large class of activation functions, namely functions of bounded variation. Rates of convergence to the optimal loss are also derived.

A fast nearest neighbor algorithm for pattern classification is proposed and tested on real data. The patterns (points in d-dimensional Euclidean space) are sorted along a space-filling curve. This way the multi-dimensional problem is compressed to the simplest case of the nearest neighbor search in one dimension. Instead of Euclidean distance a metric on space-filling curve is used. The method may be inferior or superior to the k-NN rule in multidimensional Euclidean space.

Studies convergence properties of radial basis function (RBF) networks for a large class of basis functions, and reviews the methods and results related to this topic. The authors obtain the network parameters through empirical risk minimization. The authors show the optimal nets to be consistent in the problem of nonlinear function approximation and in nonparametric classification. For the classification problem the authors consider two approaches: the selection of the RBF classifier via nonlinear function estimation and the direct method of minimizing the empirical error probability. The tools used in the analysis include distribution-free nonasymptotic probability inequalities and covering numbers for classes of functions.

Studies the convergence properties of the mean integrated absolute error (MIAE) for kernel regression estimators (KRE) and radial basis function (RBF) nets. The authors show that the MIAE of KRE and RBF nets converges to zero as the size of network and the size of training sequence tend to infinity, and the authors give the upper bound for the convergence rate for approximating functions satisfying Lipschitz condition of order /spl alpha/,0

In the paper convergence properties of radial basis function (RBF) networks are studied for a large class of basis functions. The universal approximation property of the nets is shown. Parameters of RBF nets are learned through empirical risk minimization. The optimal nets are shown to be consistent in nonparametric classification. The tools used in the analysis include Vapnik-Chervonenkis (VC) dimension and the covering numbers.

In the paper the estimation of block oriented systems is discussed. Particular attention is devoted to memoryless and dynamical systems with cascade structure. An optimal model of a memoryless cascade system is given and estimated by the kernel regression estimate. Nonlinear dynamical systems of the Hammerstein and Wiener type are estimated by means of nonparametric techniques. The convergence of estimation procedures is investigated.>

This correspondence presents a segmentation and fitting method using a new robust estimation technique. We present a robust estimation method with high breakdown point which can tolerate more than 80% of outliers. The method randomly samples appropriate range image points in the current processing region and solves equations determined by these points for parameters of selected primitive type. From K samples, we choose one set of sample points that determines a best-fit equation for the largest homogeneous surface patch in the region. This choice is made by measuring a residual consensus (RESC), using a compressed histogram method which is effective at various noise levels. After we get the best-fit surface parameters, the surface patch can be segmented from the region and the process is repeated until no pixel left. The method segments the range image into planar and quadratic surfaces. The RESC method is a substantial improvement over the least median squares method by using histogram approach to inferring residual consensus. A genetic algorithm is also incorporated to accelerate the random search.>

A new method of separating touching unconstrained handwritten digits is proposed. A binary image containing a string of touching digits is scanned to give contour chains. The chains are analyzed and subdivided into four kinds of regions: valleys, mountains, holes, and open regions. Individual points of interest in the outer contour are then identified, e.g., points of high curvature. The separating path is assumed to pass between some pair of these significant contour points (SCPs). Nine features of the SCPs are measured and are used to sort the list of all possible pairings of SCPs. Preliminary results show that the correct cut is sorted within the first three choices in 89% of tests.>

It is shown that frequency sensitive competitive learning (FSCL), one version of the recently improved competitive learning (CL) algorithms, significantly deteriorates in performance when the number of units is inappropriately selected. An algorithm called rival penalized competitive learning (RPCL) is proposed. In this algorithm, not only is the winner unit modified to adapt to the input for each input, but its rival (the 2nd winner) is delearned by a smaller learning rate. RPCL can be regarded as an unsupervised extension of Kohonen's supervised LVQ2. RPCL has the ability to automatically allocate an appropriate number of units for an input data set. The experimental results show that RPCL outperforms FSCL when used for unsupervised classification, for training a radial basis function (RBF) network, and for curve detection in digital images.

The authors randomly sample appropriate range image points and solve equations determined by these points for the parameters of selected primitive type. From K samples they measure residual consensus to choose one set of sample points that determines an equation having the best fit for the largest homogeneous surface patch in the current processing region. The residual consensus is measured by a compressed histogram method that works at various noise levels. The estimated surface patch is extracted out of the processing region to avoid further computation. A genetic algorithm is used to accelerate the search speed.>

For the classical k-means clustering algorithm, the problem of selecting an appropriate k is a hard problem and affects the performance of k-means strongly. When used for clustering analysis, the conventional competitive learning (CL) algorithms also have a similar crucial problem-the selection of an appropriate number of neural units. The performance of frequency sensitive competitive learning (FSCL)-one version of the improved CL algorithms, also significantly deteriorates when the number of units is inappropriately selected. The paper proposes a new algorithm called rival penalized competitive learning (RPCL), which has the ability of automatically allocating an appropriate number of units for an input data set. The experimental results have shown that RPCL outperforms FSCL significantly when they are used for unsupervised classification, and supervised classification through the radial basis function net.>

An improved exponential bound on the L/sub 1/ error for the recursive kernel regression estimates is derived. It is shown, using the martingale device, that weak, strong and complete L/sub 1/ consistencies are equivalent. Consequently the conditions on a certain smoothing sequence are necessary and sufficient for strong L/sub 1/ consistency of the recursive kernel regression estimate. The rates of global convergence are also given. Obtained results are applied to recursive classification rules and to nonlinear time series estimation.>

Possible solutions to the problem of combining classifiers can be divided into three categories according to the levels of information available from the various classifiers. Four approaches based on different methodologies are proposed for solving this problem. One is suitable for combining individual classifiers such as Bayesian, k-nearest-neighbor, and various distance classifiers. The other three could be used for combining any kind of individual classifiers. On applying these methods to combine several classifiers for recognizing totally unconstrained handwritten numerals, the experimental results show that the performance of individual classifiers can be improved significantly. For example, on the US zipcode database, 98.9% recognition with 0.90% substitution and 0.2% rejection can be obtained, as well as high reliability with 95% recognition, 0% substitution, and 5% rejection.>

A new modification of the subspace pattern recognition method, called the dual subspace pattern recognition (DSPR) method, is proposed, and neural network models combining both constrained Hebbian and anti-Hebbian learning rules are developed for implementing the DSPR method. An experimental comparison is made by using our model and a three-layer forward net with backpropagation learning. The results illustrate that our model can outperform the backpropagation model in suitable applications.

The exponential, distribution-free bounds for the kernel classification rule are derived. The equivalence of all modes of the global convergence of the rule is established under optimal assumptions on the smoothing sequence. Also derived is the optimal global rate of convergence of the kernel regression estimate within the class of Lipschitz distributions. The rate is optimal for the nonparametric regression, but not for classifications. It is shown. using the martingale device, that weak, strong, and complete L/sub 1/ Bayes risk consistencies are equivalent. Consequently the conditions on the smoothing sequence h/sub n/ to 0 and nh/sub n/ to infinity are necessary and sufficient for Bayes risk consistency of the kernel classification rule. The rate of convergence of the kernel classification rule is also given.>

A novel recognition system has been implemented to solve the difficult problem of handwritten numeral recognition. In this system, the Fourier descriptors are used as dominant features, and a modified backpropagation model is applied to classification. A novel backpropagation learning algorithm has been developed, and its performance has been evaluated. The results show that the learning algorithm is superior to the original backpropagation model. The proposed algorithm was able to solve the nonconvergence problem typically occurring with the standard backpropagation approach. The algorithm has been tested on handwritten numerals collected by the US Post Office

An algorithm for estimating motion parameters of a rigid body from range data is presented. The best correspondence between two three-dimensional point sets is established using a Hopfield neural network. Once the correspondence is built, the δ-bound matching concept is introduced to discard the matched noise pairs and to estimate motion parameters. The method is tolerant to noise and missing points. It is easily extended to plane and surface matching. Simulation results are given for noisy synthetic data

The estimation of a multiple-input single-output discrete Hammerstein system is studied. Such a system contains a nonlinear memoryless subsystem followed by a dynamic linear subsystem. The impulse response of the dynamic linear subsystem is obtained by the correlation method. The main results concern the estimation of the nonlinear memoryless subsystem. No conditions are imposed on the functional form of the nonlinear subsystem, and the nonlinearity is recovered using the kernel regression estimate. The distribution-free pointwise and global convergence of the estimate is demonstrated-that is, no conditions are imposed on the input distribution, and convergence is proven for virtually all nonlinearities. The rates of pointwise as well as global convergence are obtained for all input distributions and for Lipschitz type nonlinearities.>

Two kinds of Fourier shape descriptors (FDs) are considered: ZR defined by C.T. Zahn and R.S. Roskies (1972) and G defined by G.H. Granlund (1972). In the first part of the paper ZR descriptors are studied. Two modifications of ZR descriptors are proposed. The new descriptors are based on step signature and smoother signature. The amplitudes of FDs are invariant under rotations, translations, changes in size, mirror reflections, and shifts in the starting point. In all the cases the reconstruction accuracy in terms of the number of FDs is studied, resulting in approximation error bounds. An efficient reconstruction method not requiring numerical integration is proposed for polygonal shapes. In the second part of the work theoretical results are verified in numerical experiments involving handwritten characters. In the same experiments, the performances of ZR and G descriptors are compared.>

Both nonrecursive and recursive nonparametric regression estimates are studied. The rates of weak and strong convergence of kernel estimates, as well as corresponding multiple classification errors, are derived without assuming the existence of the density of the measurements. An application of the obtained results to a nonparametric Bayes predication is presented.

It is shown that the kernel estimate of the regressionE(Y|X = x)is weakly or strongly consistent for almost allx(\mu), where\muis the probability measure ofX. The result is valid for any distribution ofX. The asymptotical optimality of classification rules derived from the estimate is examined. The optimality is independent of class distributions, i.e., it is distribution-free.

It is shown that the recursive kernel estimate of the regression functionE(Y|X = x)is consistent at almost everyx(\mu)regardless of the distribution\muofX. Thus the result is distribution-free. From this we show that the risk for a suitable classification rule derived from the estimate converges to Bayes' risk, no matter what the class distributions are.