In this paper we propose a method for accelerating grid search for model selection by pruning grid points. The proposed method is called Pruning Grid Search (PGS). We use the support vector machine (SVM) with RBF kernels, which has the radius parameter for the RBF kernels and the margin parameter to control the margin of the SVM. In general, by increasing either the radius parameter or the margin parameter, the accuracy for the training data increases. In PGS, we increase either of the parameter values or decrease the radius parameter value, detect the overfitting/underfitting point by the drop of the cross-validation accuracy, and prune the grid points that are beyond the overfitting/underfitting point. We evaluate PGS for benchmark problems and show that PGS can delete grid points from evaluation without deteriorating the generalization ability.

To improve the generalization ability and to reduce output and hyperparameter variations of a classifie, in previous work, linearly separable data with large margins are selected from the training data. The usefulness of this method is tested for classifier including the support vector machine (SVM). This means that bounded support vectors are harmful to both the high generalization ability and the stableness of the SVM. In this paper, we firs discuss the problems of the selection method and examine whether the harmfulness is true. To do this, firs we train the SVM using a data set and test the generalization ability. Then we delete all the misclassifie or bounded support vectors from the data set, train the SVM with the reduced data set, and test the generalization ability. Finally, we compare the generalization abilities and the stableness for the original and reduced training data sets. The computer experiments show that in general the accuracy of the test data decreases the most by deleting all the bounded support vectors and statistical analysis shows that in general the deletion of misclassifie or bounded support vectors does not improve the generalization ability with statistical significance In addition, the stableness of hyperparameters for the deletion of misclassifie or bounded support vectors is not significant

The minimal complexity support vector machine (ML1 SVM), which is a fusion of the standard support vector machine (L1 SVM) and the minimal complexity machine (MCM), works to improve the generalization ability over the L1 SVM. In this paper, we introduce a three-kernel structure into the ML1 SVM. We call it ML1 SVM (MK). We speed up model selection by three-stage cross-validation: In the first stage, we determine the hyperparameters for the L1 SVM (MK) fixing the hyperparameters for multiple kernels. In the second stage we determine the hyperparameters for the multiple kernels, and in the third stage, we determine the hyperparameter for the ML1 SVM (MK) for controlling the maximum margin. We also discuss a method for accelerating the second stage of model selection. In the computer experiment using two-class and multiclass problems, the ML1 SVM (MK) showed statistically comparable to, or better than the L1 SVM (MK), ML1 SVM, and L1 SVM.

The minimum complexity L1 SVM (ML1 SVM) is the fusion of the minimal complexity machine (MCM) and the standard support vector machine (L1 SVM) to realize high generalization ability over the L1 SVM. The fusion is realized by minimizing the upper bound on the decision function values (i.e., minimizing the maximum margin as well as maximizing the minimum margin) in the L1 SVM. In this paper, to avoid worsening the generalization ability by minimizing the upper bound, we introduce the soft upper bound on the decision function values. This is realized by allowing some of the training data outputs to exceed the upper bound. We call this architecture soft upper-bound L1 SVM (SL1 SVM). By introducing the soft upper-bound, the number of hyperparameters increases by one. We explain how to cancel the increase by setting the hyperparameter value properly, and clarify the characteristics of the SL1 SVM. Finally, we compare the SL1 SVM with ML1 SVMs using several benchmark data sets and show that overfitting occurs for the SL1 SVM more frequently than for the ML1 SVMs.

The minimal complexity linear programming support vector machine (MLP SVM) was proposed to solve the problem of unbounded non-unique solutions of the minimal complexity machine (MCM). The MLP SVM minimizes the maximum margin that is the maximum distance between training data and the separating hyperplane as well as maximizes the minimum margin. Therefore, the generalization ability may be worsened if outliers are included and they affect the slope and the location of the separating hyperplane. To solve this problem, in this paper, we propose the soft upper-bound MLP SVM (SLP SVM), in which the outliers that affect the hyperplane are suppressed by introducing the slack variables. This introduction leads to the increase of hyperparameters. We discuss how to reduce the number of hyperparameters to speed up model selection. By computer experiments we compare the generalization ability and training time of the SLP SVM with those of the MLP SVM, MCM, and other SVM based classifiers using two-class and multiclass problems.

The minimal complexity machine (MCM) minimizes the maximum distance between training data and the separating hyperplane and is shown to generalize better than the conventional support vector machine. In this paper, we analyze the MCM and clarify the conditions that the solution of MCM is nonunique and unbounded. To resolve the unboundedness, we propose the minimal complexity linear programming support vector machine (MLP SVM), in which the minimization of the maximum distance between training data and the separating hyperplane is added to the linear programming support vector machine (LP SVM). By computer experiments we show that the solution of the MCM is unbounded under some conditions and that the MLP SVM generalizes better than the LP SVM for most of the two-class and multiclass problems.

In this paper, we train support vector regressors (SVRs) fusing sequential minimal optimization (SMO) and Newton's method. We use the SVR formulation that includes the absolute variables. A partial derivative of the absolute variable with respect to the associated variable is indefinite when the variable takes on zero. We determine the derivative value according to whether the optimal solution exits in the positive region, negative region, or at zero. In selecting working set, we use the method that we have developed for the SVM, namely, in addition to the pair of variables selected by SMO, loop variables that repeatedly appear in training, are added to the working set. By this method the working set size is automatically determined. We demonstrate the validity of our method over SMO using several benchmark data sets.

In our previous work, we proposed feature selection by block addition (BA) and block deletion (BD). In this paper, to further reduce features, we iterate BABD until no features are eliminated. In our method, we add several features at a time to the feature set until a stopping condition is satisfied. Then we delete features that do not deteriorate the selection criterion by block deletion. We iterate block addition and block deletion for the selected feature set until no features are eliminated. By computer experiments using micro array data sets we show that for some micro array data sets, features are further deleted by iterating BABD and as the selection and ranking criteria the weighted sum of the recognition error rate and the average of margin errors is better than the recognition error rate in obtaining a feature set with high generalization ability.

In this paper, we propose two methods for subspace based support vector machines (SS-SVMs) which are subspace based least squares support vector machines (SSLS-SVMs) and subspace based linear programming support vector machines (SSLP-SVMs): 1) optimum selection of the dictionaries of each class subspace from the standpoint of classification separability, and 2) speeding up training SS-SVMs. In method 1), for SSLS-SVMs, we select the dictionaries with optimized weights, and for SSLP-SVMs, we select the dictionaries without non-negative constraints. In method 2), the empirical feature space is obtained by using only the training data belonging to a class instead of using all the training data. Thus the dimension of the empirical feature space and training cost become lower. We demonstrate the effectiveness of the proposed methods over the conventional method for two-class bench mark datasets.

In our previous work we have developed an active set training method of L2 support vector machines (SVMs) using Newton's method. Because the method allows a solution to be infeasible during training, convergence of the method is not guaranteed. In this paper, we guarantee convergence of active set training by limiting the corrections under the constraints when slow convergence is detected. Namely, we start training the L2 SVM with a subset of training data, delete non-positive dual variables from the working set as well as the variables with margins larger than or equal to 1, add violating variables to the working set, and repeat training. We monitor the number of violation fluctuations and if it exceeds a specified value, we obtain a feasible solution prohibiting addition of violating variables. Then for a feasible solution, we start active set training limiting the corrections within a feasible solution. By computer experiments, we show that the proposed training method is faster and more stable than the previous method.

In this paper, we propose a novel learning framework to reformulate a kernel-based classifier in terms of three modular components: kernel-function determination to incorporate domain knowledge, sparse data representation using FastMap and its variants, and supervised classification performed by using primal form analyzers such as linear SVM. The first important property of this approach is the reusability of the modules: Each module can be easily replaced by its counterparts for a specific learning purpose, e.g., the sparse representation of the data can not only support classification tasks but also be applied in function regression or unsupervised data analysis. Another contribution of the proposed approach is that it enables easy adaption of available primal-form algorithms for nonlinear kernel-based learning. Finally, numerical experiments show that FastMap and SupFM can yield efficient sparse representations with nonlinear kernels. The representation realized better sparsity while maintaining a generalization ability that is comparable to that of the regular SVM classifier.

In this paper, we discuss subspace based least squares support vector machines (SSLS-SVMs), in which an input vector is classified into the class with the maximum similarity. Namely, we define the similarity measure for each class by the weighted sum of vectors called dictionaries and optimize the weights so that the margin between classes is optimized. Because the similarity measure is defined for each class, the similarity measure associated with a data sample needs to be the largest among all the similarity measures. Introducing slack variables we define these constraints by equality constraints. Then the proposed SSLS-SVMs is similar to LS-SVMs by all-at-once formulation. Because all-at-once formulation is inefficient, we also propose SSLS-SVMs by one-against-all formulation. We demonstrate the effectiveness of the proposed methods with the conventional method for two-class problems.

Support vector regressors (SVRs) usually give sparse solutions but as a regression problem becomes more difficult the number of support vectors increases and thus sparsity is lost. To solve this problem, in this paper we propose sparse support vector regressors (S-SVRs) trained in the reduced empirical feature space. First by forward selection we select the training data samples, which minimize the regression error estimated by kernel least squares. Then in the reduced empirical feature space spanned by the selected, mapped training data, we train the SVR in the dual form. Since the mapped support vectors obtained by training the S-SVR are expressed by the linear combination of the selected, mapped training data, the support vectors, in the sense that form a solution, are selected training data. By computer simulation, we compare performance of the proposed method with that of the regular SVR and that of the sparse SVR based on Cholesky factorization.

In subspace methods, the subspace associated with a class is represented by a small number of vectors called dictionaries and using the dictionaries the similarity measure is defined and an input is classified into the class with the highest similarity. Usually, each dictionary is given an equal weight. But if subspaces of different classes overlap, the similarity measures for the overlapping regions will not give useful information for classification. In this paper, we propose optimizing the weights for the dictionaries using the idea of support vector machines (SVMs). Namely, first we map the input space into the empirical feature space, perform kernel principal component analysis (KPCA) for each class, and define a similarity measure. Then considering that the similarity measure corresponds to the hyperplane, we formulate the optimization problem as maximizing the margin between the class associated with the dictionaries and the remaining classes. The optimization problem results in all-at-once formulation of linear SVMs. We demonstrate the effectiveness of the proposed method with that of the conventional methods for two-class problems.

We propose a feature selection criterion based on kernel discriminant analysis (KDA) for a n-class problem, which finds eigenvectors on which the projected class data are locally maximally separated. The proposed criterion is the sum of the objective function values of KDA associated with the n-1 eigenvectors. The criterion results in calculating the sum of n-1 eigenvalues associated with the eigenvectors and is shown to be monotonic for the deletion or addition of features. Using the backward feature selection strategy, for several multi-class data sets, we evaluated the proposed criterion and the criterion based on the recognition rate of the support vector machine (SVM) evaluated by cross-validation. From the standpoint of generalization ability the proposed criterion is comparable with the SVM-based recognition rate, although the proposed method does not use cross-validation.

We discuss sparse support vector machines (sparse SVMs) trained in the reduced empirical feature space. Namely, we select the linearly independent training data by the Cholesky factorization of the kernel matrix, and train the SVM in the dual form in the reduced empirical feature space. Since the mapped linearly independent training data span the empirical feature space, the linearly independent training data become support vectors. Thus if the number of linearly independent data is smaller than the number of support vectors trained in the feature space, sparsity is increased. By computer experiments we show that in most cases we can reduce the number of support vectors without deteriorating the generalization ability.

In function approximation, if datasets have many redundant input variables, various problems such as deterioration of the generalization ability and an increase of the computational cost may occur. One of the methods to solve these problems is variable selection. In pattern recognition, the effectiveness of backward variable selection by block deletion is shown. In this paper, we extend this method to function approximation. To prevent the deterioration of the generalization ability, we use the approximation error of a validation set as the selection criterion. And to reduce computational cost, during variable selection we only optimize the margin parameter by cross-validation. If block deletion fails we backtrack and start binary search for efficient variable selection. By computer experiments using some datasets, we show that our method has performance comparable with that of the conventional method and can reduce computational cost greatly. We also show that a set of input variables selected by LS-SVRs can be used for SVRs without deteriorating the generalization ability.

In this paper, a feature extraction method for online classification problems is proposed by extending kernel principal component analysis (KPCA). In our previous work, we proposed an incremental KPCA algorithm which could learn a new input incrementally without keeping all the past training data. In this algorithm, eigenvectors are represented by a linear sum of linearly independent data which are selected from given training data. A serious drawback of the previous IKPCA is that many independent data are prone to be selected during learning and this causes large computation and memory costs. For this problem, we propose a novel approach to the selection of independent data; that is, they are not selected in the high-dimensional feature space but in the low-dimensional eigenspace spanned by the current eigenvectors. Using this method, the number of independent data is restricted to the number of eigenvectors. This restriction makes the learning of the modified IKPCA (M-IKPCA) very fast without loosing the approximation accuracy against true eigenvectors. To verify the effectiveness of M-IKPCA, the learning time and the accuracy of eigenspaces are evaluated using two UCI benchmark datasets. As a result, we confirm that the learning of M-IKPCA is at least 5 times faster than the previous version of IKPCA.

In this paper we address the problem of how to implement a multi-class classifier by an ensemble of one-class classifiers. One-class classifiers are first trained for each class and then a decision function is formulated based on minimum distance rules. Two kinds of one-class classifiers are explored: the Support Vector Domain Description and a Kernel Principle Component Analysis based method. Both of the two methods can work in the feature space and deal with nonlinear classification problems. Experiments on some benchmark datasets show that the proposed methods with carefully tuned parameters have comparable generalization ability with Support Vector Machines while having some other advantages.

In this paper, we propose an incremental learning model for ensemble classifier systems. In the proposed model, the number of classifiers is predetermined and fixed during the learning, and all classifiers are updated at every learning stage based on an extended algorithm of AdaBoost.M1. A neural network model called Resource Allocating Network with Long-Term Memory (RAN-LTM), which has been developed to realize stable incremental learning, is adopted as a classifier. We also propose a new method to update the classifier weights in the weighted majority voting under the one-pass incremental learning situations. In the experiments, first we verify that the proposed model can learn incrementally without serious forgetting and that the performance is not influenced seriously by the size of a training subset given at every learning stage. Then, through a comparison with Resource Allocating Network (RAN), RAN-LTM, and AdaBoost.M1, we demonstrate that the proposed incremental ensemble classifier system has comparable performance with a batch-learning ensemble classifier system, and that it outperforms both both-learning and incremental-learning single-classifier systems.

In this paper we present a novel feature selection algorithm for SVMs which works by estimating the stability of a feature's contribution to some evaluation criterion. This algorithm is extremely fast as only a small number of SVM classifiers need to be trained for feature selection. Robust results are shown with toy as well as real-life datasets. Furthermore, we combine this method with a backward elimination procedure. The combined algorithm performs stably and shows optimal performance compared with other feature selection methods. Another merit of the combined algorithm is that it can estimate the optimal number of features with the best prediction power. This method is applicable to both linear and nonlinear problems.

In this paper, a new approach to face recognition is presented in which not only a classifier but also a feature space of input variables is learned incrementally to adapt to incoming training samples. A benefit of this type of incremental learning is that the search for useful features and the learning of an optimal decision boundary are carried out in an online fashion. To implement this idea, an extended version of incremental principal component analysis (IPCA) and resource allocating network with long-term memory (RAN-LTM) are effectively combined. Using IPCA, a feature space is updated by rotating its eigen-axes and increasing the dimensions to adapt to a new training sample. In RAN-LTM, a small number of training samples called memory items are selected and they are utilized for retraining a classifier to realize an excellent incremental ability. To accommodate the classifier to the evolution of the feature space, we present a way to reconstruct the neural classifier without keeping all of the training samples given previously. In the experiments, the proposed incremental learning model is evaluated over a self-compiled face image database. As the result, we verify that the proposed model works well without serious forgetting and the test performance is improved as the learning stages proceed.

In the conventional incremental training of support vector machines (SVMs), candidates of support vectors tend to be deleted if the separating hyperplane rotates as the training data are added. To solve this problem, in this paper, we propose an incremental training method using one-class support vector machines. First, we generate a hypersphere for each class. Then, we keep data that exist near the boundary of the hypersphere as candidates of support vectors and delete others. By computer simulations for two-class benchmark data sets, we show that we can robustly delete data considerably without deteriorating the generalization ability.

We propose a novel spatially chunking algorithm to speed up the support vector clustering (SVC) method for large data sets. The input data set is first divided into subsets where samples are geometrically adjacent to each other, an SVC is trained for each subset, and finally the clustering results of the local SVCs are combined to yield a global clustering solution. This method can save the computation cost for SVC by breaking the quadratic programming problem into smaller ones, and since parameter selection is done for each subset, it is able to deal with unevenly distributed data sets. The proposed method has demonstrated satisfactory performance with image segmentation problems on both gray scale and color images.

In this paper, first we discuss acceleration of classification by reducing support vectors. Then, we discuss multiclass least squares SVMs (LS-SVMs) that resolve unclassifiable regions for multiclass problems: fuzzy one-against-all LS-SVMs, fuzzy pairwise LS-SVMs, and all-at-once LS-SVMs. Next, we compare the three types of LS-SVMs from the standpoint of training difficulty and show that the fuzzy one-against-all LS-SVM and the all-at-once LS-SVM have similar decision boundaries when classification problems are linearly separable in the feature space. Finally, we evaluate three types of multiclass LS-SVMs for some benchmark data sets and show that classification performance of fuzzy one-against-all and one-against-all LS-SVMs are almost the same but inferior to that of fuzzy pairwise LS-SVMs.

In this paper, we discuss a kernel version of fuzzy classifiers with ellipsoidal regions to improve generalization ability. First, we map the input space into the implicit feature space induced by a kernel function. Then we generate a fuzzy rules in the feature space and tune the slopes of membership functions successively until there is no improvement in the recognition rate of the training data. We evaluate our method using numeral and hiragana data of vehicle license plates, and blood cell data. Except for the numeral data, the generalization ability is improved against that of the conventional fuzzy classifier with ellipsoidal regions, and is comparable to that of support vector machines.

In this paper, we compare L1 and L2 support vector machines from the standpoint of training time and the generalization ability. The generalization ability for seven benchmark data sets are almost the same but training time of L1-SVMs is usually shorter than that of L2-SVMs. We also compare the effect of the approximate KKT (Karush-Kuhn-Tucker) conditions using the bias term and the exact KKT conditions. According to the computer experiments, since the approximate KKT conditions give a conservative estimate of violating variables, training time using the approximate KKT conditions is usually shorter.

To avoid the catastrophic interference in incremental learning, we have proposed resource allocating network with long term memory (RAN-LTM). In RAN-LTM, not only a new training sample but also some memory items stored in long-term memory are trained based on a gradient descent algorithm is usually slow and can be easily fallen into local minima. To solve these problems, we propose a fast incremental learning algorithm of RAN-LTM, in which its centers are not trained but selected based on output errors. This model does not need so much memory capacity and it is also realizes robust incremental learning ability. To verify these characteristics of RAN/spl I.bar/LTM, we apply it to two function approximation problems: one-dimensional function approximation and prediction of Mackey-Glass time series. From the experimental results, it is verified that the proposed RAN-LTM can learn fast and accurately without large main memory unless incremental learning is conducted over long period of time.

In least squares support vector machines (LS-SVMs), the optimal separating hyperplane is obtained by solving a set of linear equations instead of solving a quadratic programming problem. But since SVMs and LS-SVMs are formulated for two-class problems, unclassifiable regions exist when they are extended to multiclass problems. In this paper, we discuss fuzzy least squares support vector machines that resolve unclassifiable regions for multiclass problems. We define a membership function in the direction perpendicular to the optimal separating hyperplane that separates a pair of classes. Using the minimum or average operation for these membership functions, we define a membership function for each class. Using some benchmark data sets, we show that recognition performance of fuzzy LS-SVMs with the minimum operator is comparable to that of fuzzy SVMs, but fuzzy LS-SVMs with the average operator showed inferior performance.

In a fuzzy classifier with ellipsoidal regions, each cluster is approximated by a center and a covariance matrix, and the membership function is calculated using the inverse of the covariance matrix. Thus, when the number of training data is small, the covariance matrix becomes singular and the generalization ability is degraded. In this paper, during the symmetric Cholesky factorization of the covariance matrix, if the input of the square root is smaller than a prescribed positive value, we replace the input with the prescribed value. Further, we tune the slopes of the membership functions so that the margins are maximized. We show the validity of our method by computer simulations.

In this paper we discuss a fuzzy classifier with pyramidal membership functions and discuss a performance improvement by changing linear membership functions to quadratic membership functions. First we define the classifier with quadratic membership functions and then discuss the training method that maximises the recognition rate by counting the net increase in the recognition rate by changing the slope and the center of each membership function. Finally, we demonstrate the recognition improvement by quadratic membership functions for the iris and blood cell data sets.

In the previous work by Abe et al. (1998), the backward feature selection method based on class regions approximated by ellipsoids was developed. In this paper, we accelerate feature selection by the forward selection search, the symmetric Cholesky factorization, and deletion of duplicated calculations between consecutive factorizations. The feature selection for two data sets shows that our method is faster than and as robust as the previous method.

Discusses robust function approximation when the Takagi-Sugeno type model is used for the consequent part of fuzzy rules. With this model, the parameters of the linear equation that defines the output value of the fuzzy rule are determined by the least-squares method. Therefore, if the training data include outliers, the method fails to determine the parameter values correctly. To overcome this problem we use the least median of squares method. Among the original training data set, we randomly select training data more than the number of parameters, and determine the parameter values using the least-squares method. We repeat this many times and determine the parameters with the smallest median of squared errors. We compare the proposed method with the least-squares method and the conventional least median of squares method using the data generated by the Mackey-Glass differential equation.

We discuss training of three-layer neural network classifiers by solving inequalities. We first represent each class by the center of the training data belonging to the class, and determine the set of hyperplanes that separate each class into a single region. Then, according to whether the center is on the positive or negative side of the hyperplane, we determine the target values of each class for the hidden neurons. Since the convergence condition of the neural network classifier is now represented by the two sets of inequalities, we solve the sets successively by the Ho-Kashyap algorithm. We demonstrate the advantage of our method over the BP using three benchmark data sets.

This paper deals with identical parallel machine scheduling problems with two kinds of objective functions, i.e., both regular and non-regular objective functions, and proposes a genetic algorithm approach in which (a) the sequence of jobs on each machine as well as the assignment of jobs to machines are determined directly by referring to a string (genotype), and (b) the start time of each job is fixed by solving the linear programming problem and a feasible schedule (phenotype) is obtained. As for (b), we newly introduce a method of representing the problem to determine the start time of each job as a linear programming problem whose objective function is formed as a weighted sum of the original multiple objective functions. This method enables us to obtain a lot of potential schedules. Moreover, through computational experiments by using our genetic algorithm approach, the effectiveness for generating a variety of Pareto-optimal schedules is investigated.

This paper discusses two types of fuzzy function approximators that dynamically generate fuzzy rules with ellipsoidal regions: a function approximator based on Takagi-Sugeno type model with the center-of-gravity defuzzification and a function approximator based on a radial basis function network. Hereafter the former is called FACG and the latter is called FALC. In FACG, for each training datum the number of the training data that are within the specified distance is calculated and the training datum which has the maximum number of the training data is selected as the center of a fuzzy rule and the covariance matrix is calculated using the training data around the center. Then the parameters of the linear equation that defines the output value of the fuzzy rule are determined by the least-squares method using the training data around the center. In FALC, the training datum with the maximum approximation error is selected as the center of a fuzzy rule. Then using the training data around the center, the covariance matrix is calculated, and the parameters of a linear equation that determines the output value are calculated by the least-squares method. Performance of FACG and FALC is compared with that of multilayered neural networks and other fuzzy function approximators for the data generated by the Mackey-Glass differential equation and the data from a water purification plant.

In our previous work, we developed a fuzzy classifier with ellipsoidal regions that has a training capability. In this paper, we extend the fuzzy classifier to diagnosis problems, in which the training data belonging to abnormal classes are difficult to obtain while the training data belonging to normal classes are easily obtained. Assuming that there are no data belonging to abnormal classes, we first train the fuzzy classifier with only the data belonging to normal classes. We then introduce the threshold of the minimum-weighted distance from the centers of the clusters for the data belonging to normal classes. If the unknown data is within the threshold, we classify the data into normal classes and, if not, abnormal classes. The operator checks whether the diagnosis is correct. If the incoming data is classified into the same normal class both by the classifier and the operator, nothing is done. But if the input data is classified into the different normal classes by the classifier and the operator, or if the incoming data is classified into an abnormal class, but the operator classified it into a normal class, the slopes of the membership functions of the fuzzy rules are tuned. If the operator classifies the data into an abnormal class, the classifier is retrained adding the newly obtained data irrespective of the classifier's classification result. The online training is continued until a sufficient number of the data belonging to abnormal classes are obtained. Then the threshold is optimized using the data belonging to both normal and abnormal classes. We evaluate our method using the Fisher iris data, blood cell data, and thyroid data, assuming some of the classes are abnormal.

We present an efficient method for extracting fuzzy rules directly from numerical input-output data for function approximation problems. First, we convert a given function approximation problem into a pattern classification problem. This is done by dividing the universe of discourse of the output variable into multiple intervals, each regarded as a class, and then by assigning a class to each of the training data according to the desired value of the output variable. Next, we partition the data of each class in the input space to achieve a higher accuracy in approximation of class regions. Partition terminates according to a given criterion to prevent excessive partition. For class region approximation, we discuss two different types of representations using hyperboxes and ellipsoidal regions, respectively. Based on a selected representation, we then extract fuzzy rules from the approximated class regions. For a given input datum, we convert, or in other words, defuzzify, the resulting vector of the class membership degrees into a single real value. This value represents the final result approximated by the method. We test the presented method on a synthetic nonlinear function approximation problem and a real-world problem in an application to a water purification plant. We also compare the presented method with a method based on neural networks.

We discuss a fuzzy classifier with ellipsoidal regions that dynamically generates clusters. First, for the data belonging to a class we define a fuzzy rule with an ellipsoidal region. Then we tune the fuzzy rules successively until there is no improvement in the recognition rate of the training data. Then if the number of the data belonging to a class that are misclassified into another class exceeds a prescribed number, we define a new cluster to which those data belong and the associated fuzzy rule. Then we tune the newly defined fuzzy rules in the similar way as stated above, fixing the already obtained fuzzy rules. We iterate the generation of clusters and tuning of the newly generated fuzzy rules until the number of the data belonging to a class that are misclassified into another class does not exceed the prescribed number. We evaluate our method using thyroid data and blood cell data.

This paper gives introduction to four knowledge acquisition systems that use hyperboxes to represent knowledge. Hyperbox representation is interesting because it can uniformly represent both knowledge acquired by human experts and knowledge automatically learned from training examples. Potential applications of these systems are described. Discussions on performance issues and on future research topics are given.

In this paper, we discuss a fuzzy classifier with ellipsoidal regions that dynamically generates clusters. First, for the data belonging to a class we define a fuzzy rule with an ellipsoidal region. Namely, using the training data for each class, we calculate the center and the covariance matrix of the ellipsoidal region for the class. Then we tune the fuzzy rules, i.e., the slopes of the membership functions, successively until there is no improvement in the recognition rate of the training data. Then if the number of the data belonging to a class that are misclassified into another class exceeds a prescribed number, we define a new cluster to which those data belong and the associated fuzzy rule. Then we tune the newly defined fuzzy rules in the similar way as stated above, fixing the already obtained fuzzy rules. We iterate generation of clusters and tuning of the newly generated fuzzy rules until the number of the data belonging to a class that are misclassified into another class does not exceed the prescribed number. We evaluate our method using thyroid data, Japanese Hiragana data of vehicle license plates, and blood cell data. By dynamic cluster generation, the generalization ability of the classifier is improved and the recognition rate of the fuzzy classifier for the test data is the best among the neural network classifiers and other fuzzy classifiers if there are no discrete input variables.

In their previous work, the authors have developed a method for selecting features based on the analysis of class regions approximated by hyperboxes. They select features analyzing class regions approximated by ellipsoids. First, for a given set of features, each class region is approximated by an ellipsoid with the center and the covariance matrix calculated by the data belonging to the class. Then, similar to their previous work, the exception ratio is defined to represent the degree of overlaps in the class regions approximated by ellipsoids. From the given set of features, they temporally delete each feature, one at a time, and calculate the exception ratio. Then, the feature whose associated exception ratio is the minimum is deleted permanently. They iterate this procedure while the exception ratio or its increase is within a specified value by feature deletion. The simulation results show that the current method is better than the principal component analysis (PCA) and performs better than the previous method, especially when the distributions of class data are not parallel to the feature axes.

In this paper, we discuss a fuzzy classifier with ellipsoidal regions which has a learning capability. First, we divide the training data for each class into several clusters. Then, for each cluster, we define a fuzzy rule with an ellipsoidal region around a cluster center. Using the training data for each cluster, we calculate the center and the covariance matrix of the ellipsoidal region for the cluster. Then we tune the fuzzy rules, i.e., the slopes of the membership functions, successively until there is no improvement in the recognition rate of the training data. We evaluate our method using the Fisher iris data, numeral data of vehicle license plates, thyroid data, and blood cell data. The recognition rates (except for the thyroid data) of our classifier are comparable to the maximum recognition rates of the multilayered neural network classifier and the training times (except for the iris data) are two to three orders of magnitude shorter.

This paper presents a novel approach to feature selection based on analysis of class regions which are generated by a fuzzy classifier. A measure for feature evaluation is proposed and is defined as the exception ratio. The exception ratio represents the degree of overlaps in the class regions, in other words, the degree of having exceptions inside of fuzzy rules generated by the fuzzy classifier. It is shown that for a given set of features, a subset of features that has the lowest sum of the exception ratios has the tendency to contain the most relevant features, compared to the other subsets with the same number of features. An algorithm is then proposed that performs elimination of irrelevant features. Given a set of remaining features, the algorithm eliminates the next feature, the elimination of which minimizes the sum of the exception ratios. Next, a terminating criterion is given. Based on this criterion, the proposed algorithm terminates when a significant increase in the sum of the exception ratios occurs due to the next elimination. Experiments show that the proposed algorithm performs well in eliminating irrelevant features while constraining the increase in recognition error rates for unknown data of the classifiers in use.

In our previous work we have clarified global convergence of the Hopfield neural network and showed, by computer simulations, improvement of solution quality by gradually decreasing the diagonal elements of the coefficient matrix. In this paper, to accelerate convergence of the Hopfield network, at each time step the integration step size is determined dynamically so that at least one component of a variable vector reaches the surface of the hypercube. The computer simulation for the traveling salesman problem and an LSI module placement problem shows that convergence is stabilized and accelerated compared to integration by a constant step size.

In this paper, we discuss a new method for extracting fuzzy rules directly from numerical input-output data for pattern classification. Fuzzy rules with variable fuzzy regions are defined by activation hyperboxes which show the existence region of data for a class and inhibition hyperboxes which inhibit the existence of data for that class. These rules are extracted from numerical data by recursively resolving overlaps between two classes. Then, optimal input variables for the rules are determined using the number of extracted rules as a criterion. The method is compared with neural networks using the Fisher iris data and a license plate recognition system for various examples.>

In our previous work (1993) we developed a method for extracting fuzzy rules directly from numerical input-output data for pattern classification. In this paper we extend the method to function approximation. For function approximation, first, the universe of discourse of an output variable is divided into multiple intervals, and each interval is treated as a class. Then the same as for pattern classification, using the input data for each interval, fuzzy rules are recursively defined by activation hyperboxes which show the existence region of the data for the interval and inhibition hyperboxes which inhibit the existence region of data for that interval. The approximation accuracy of the fuzzy system derived by this method is empirically studied using an operation learning application of a water purification plant. Additionally, we compare the approximation performance of the fuzzy system with the function approximation approach based on neural networks.>

In this paper, a new technique for generating fuzzy rules for pattern classification is discussed. First, separation hyperplanes for classes are extracted from a trained neural network. Then, for each class, convex existence regions in the input space are approximated by shifting these hyperplanes in parallel using the training data set for the classes. Using fuzzy rules defined for each class, input data are directly classified without the use of the neural network. This method is applied to a number recognition system as well as to a blood cell classification system. Classifying performance is compared with that obtained with neural networks.>

The back-propagation algorithm is widely used for learning weights of multilayered neural networks. The major drawbacks however, are the slow convergence and lack of a proper way to set the number of hidden neurons. The author proposes a learning algorithm which solves the above problems. The weights between two layers are successively calculated, fixing other weights so that the error function, which is the square sum of the difference between the training data and the network outputs is minimised. Since the calculation results in solving a set of linearized equations, redundancy of the hidden neurons is judged by the singularity of the corresponding coefficient matrix. For the exclusive-OR and parity check circuits, excellence convergence characteristics are obtained, and the redundancy of the hidden neurons is checked by the singularity of the matrix

In a previous work, the author introduced inequality constraints into the Hopfield neural networks and clarified how to determine weights in the energy function. The author presently compares his method with the slack neuron method and shows that his method is superior because weights can be determined so that solutions not satisfying inequality constraints become unstable. Taking then-queen problem as an example, the convergence to the optimal solution is studied. It is shown that for 4- and 5-queen problems, the selection of initial values around the line segment connecting the origin and (1, . . ., 1) almost always gives the optimal solution. For the 6-queen problem, however, the above selection shows poor convergence to the optimal solution. The reason is that for a small-sized problem, the region of initial values where the network converges to the optimal solution includes the neighborhood of the center of the hypercube. But as the size becomes larger, it does not. It is also shown that the convergence of the 6-queen problem is drastically improved by the introduction of a dummy objective function

To realize the highest performance possible for a sequential processor, and to realize utilization of a large amount of existing software, an integrated Prolog processor (IPP) and its optimized compiler are now being developed.