Possibilistic c-means (PCM) clustering began in 1993, and has been used since then in many applications. In this article we discuss the geometric foundations of PCM and introduce a new hard possibilistic c-means (HPCM) clustering algorithm. We use limit theory to prove that the extended set of possibilistic c-partitions is the unit hypercube in R c n ; and that its vertices are exactly the hard possibilistic c-partitions on n objects defined herein. This enables completion of the geometric description of the domain of possibilistic clustering algorithms. We give examples that compare the results of clustering with Hard c-means (HCM) to HPCM on three small synthetic data sets. Our proof-of-concept examples show that the new algorithm performs as expected, and provides much more realistic interpretation of clusters than HCM when the data contain bridge points or noise.

The widespread use of Internet-of-Things (IoT) technologies, smartphones, and social media services generates huge amounts of data streaming at high velocity. Automatic interpretation of these rapidly arriving data streams is required for the timely detection of interesting events that usually emerge in the form of clusters. This article proposes a new relative of the visual assessment of the cluster tendency (VAT) model, which produces a record of structural evolution in the data stream by building a cluster heat map of the entire processing history in the stream. The existing VAT-based algorithms for streaming data, called inc-VAT/inc-iVAT and dec-VAT/dec-iVAT, are not suitable for high-velocity and high-volume streaming data because of high memory requirements and slower processing speed as the accumulated data increases. The scalable iVAT (siVAT) algorithm can handle big batch data, but for streaming data, it needs to be (re)applied everytime a new datapoint arrives, which is not feasible due to the associated computation complexities. To address this problem, we propose an incremental siVAT algorithm, called inc-siVAT, which deals with the streaming data in chunks. It first extracts a small size smart sample using an intelligent sampling scheme, called maximin random sampling (MMRS), then incrementally updates the smart sample points on the fly, using our novel incremental MMRS (inc-MMRS) algorithm, to reflect changes in the data stream after each chunk is processed, and finally, produces an incrementally built iVAT image of the updated smart sample, using the inc-VAT/inc-iVAT and dec-VAT/dec-iVAT algorithms. These images can be used to visualize the evolving cluster structure and for anomaly detection in streaming data. Our method is illustrated with one synthetic and four real datasets, two of which evolve significantly over time. Our numerical experiments demonstrate the algorithm's ability to successfully identify anomalies and visualize changing cluster structure in streaming data.

This article is a position paper about models and algorithms that are generally called “stream clustering.” Semantics and methods used in this field are often co-opted from static clustering, but they do not serve well for streaming data analysis. Most “state-of-the-art” methods, such as sequential k-means, Birch, CluStream, DenStream, etc., acknowledge that the data are seen but once in real streaming analysis (e.g., intrusion detection, voter fraud, etc.). Interpretation of their outputs generally overlooks the fact that when the data cannot be saved, batch clustering ideas, such as preclustering assessment, partitioning, and cluster validity are not relevant. But in the current literature, the data, or some subset of it, are often saved for hindsight evaluation (we call this fake stream clustering). Our position? Useful analysis of real streaming data is in its infancy. We do not argue that current approaches to streaming clustering are wrong: rather, we regard them as transitional methods which will eventually lead to a new and useful paradigm for this type of computation. We think that this class of models and algorithms are actually classifiers, but with a special added component, viz., continuously updated cluster footprints of the instream processing. We need to carefully define the objectives of streaming analysis, and then choose terminology and methods that suit this evolving paradigm.

To apply clustering algorithms to big data, or to build clustering ensembles, it is a standard process to sample the original data set in a way that hopefully spans the original distribution. There are at least six ways to initialize the Maximin (MM) sampling algorithm. This paper contains experiments to determine whether samples produced by the six methods differ significantly; and whether they are superior to simple random sampling. Empirical evidence supports two conclusions. First, there is not enough difference in MM samples generated by the six initializations to support using any but the least costly method: viz., using the first sample in the data as the first MM point. Second, unless the input data have subsets (clusters) that are compact and separated in a well-defined sense, random sampling is demonstrably superior to MM sampling for even small data sets.

In state-of-the-art deep learning, centralized deep learning forces end devices to pool their data in the cloud in order to train a global model on the joint data, while distributed deep learning requires a parameter server to mediate the training process among multiple end devices. However, none of these architectures scale gracefully to large-scale privacy and time-sensitive IoT applications. Therefore, we are motivated to propose a FOg-based pRivacy prEServing dEep lEarNing framework named FORESEEN, so as to achieve scalable, accurate yet private analytics. In FORESEEN, the intermediate fog nodes and the cloud collaboratively perform noisy training of deep neural networks (DNNs), while each end device and its connected fog node collaboratively perform fast, private yet accurate inference. To enhance robustness and ensure privacy, we put forward a collaborative noisy training algorithm and develop a novel representation perturber to perturb the extracted features by combining random projection, random noise addition and data nullification. To meet the required constraints of accuracy, memory and energy in IoT end devices, we build deep models with mixed-precision. Through these sophisticated designs, FORESEEN is able to not only preserve privacy but also maintain comparable inference performance. Extensive experimental results under different datasets, different inference schemes and different noise addition strategies validate the effectiveness of FORESEEN. Moreover, FORESEEN is capable of reducing the communication cost and providing inherent support for robustness and scalability.

We present a new unsupervised dimensionality reduction technique, called LN-SNE, for anomaly detection. LN-SNE generates a parametric embedding by means of Restricted Boltzmann Machines and uses a heavy-tail distribution to project data to a lower dimensional space such that dissimilarities between normal data and anomalies are preserved or strengthened. We compare LN-SNE to several benchmark dimensionality reduction methods on real datasets. The results suggest that LN-SNE for anomaly detection is less sensitive to the dimension of the latent space than the other methods and outperforms them in terms of accuracy. We empirically show that our technique scales near-linearly with respect to the number of dimensions and data size.

Cluster analysis is used to explore structure in unlabeled batch data sets in a wide range of applications. An important part of cluster analysis is validating the quality of computationally obtained clusters. A large number of different internal indices have been developed for validation in the offline setting. However, this concept cannot be directly extended to the online setting because streaming algorithms do not retain the data, nor maintain a partition of it, both needed by batch cluster validity indices. In this paper, we develop two incremental versions (with and without forgetting factors) of the Xie-Beni and Davies-Bouldin validity indices, and use them to monitor and control two streaming clustering algorithms (sk-means and online ellipsoidal clustering), In this context, our new incremental validity indices are more accurately viewed as performance monitoring functions. We also show that incremental cluster validity indices can send a distress signal to online monitors when evolving structure leads an algorithm astray. Our numerical examples indicate that the incremental Xie-Beni index with a forgetting factor is superior to the other three indices tested.

Dunn's internal cluster validity index is used to assess partition quality and subsequently identify a “best” crisp partition of n objects. Computing Dunn's index (DI) for partitions of n p-dimensional feature vector data has quadratic time complexity O(pn2), so its computation is impractical for very large values of n. This note presents six methods for approximating DI. Four methods are based on Maximin sampling, which identifies a skeleton of the full partition that contains some boundary points in each cluster. Two additional methods are presented that estimate boundary points associated with unsupervised training of one class support vector machines. Numerical examples compare approximations to DI based on all six methods. Four experiments on seven real and synthetic data sets support our assertion that computing approximations to DI with an incremental, neighborhood-based Maximin skeleton is both tractable and reliably accurate.

In current deep learning models, centralized architecture forces participants to pool their data to the central Cloud to train a global model, while distributed architecture requires a parameter server to mediate the training process. However, privacy issues, response delays, and computation and communication bottlenecks prevent these architectures from working well at the scale of Internet of Things devices. To counter these problems, in this paper we build a Fog-embedded privacy-preserving deep learning framework (FPPDL), which moves computation from the centralized Cloud to Fog nodes near the end devices. The experimental results on benchmark image datasets under different settings demonstrate that FPPDL achieves comparable accuracy to the centralized stochastic gradient descent (SGD) framework, and delivers better accuracy than the standalone SGD framework. Our evaluations also show that both computation and communication cost are greatly reduced by FPPDL, hence achieving the desired tradeoff between privacy and performance.

Trajectory prediction (TP) is of great importance for a wide range of location-based applications in intelligent transport systems, such as location-based advertising, route planning, traffic management, and early warning systems. In the last few years, the widespread use of GPS navigation systems and wireless communication technology enabled vehicles has resulted in huge volumes of trajectory data. The task of utilizing these data employing spatio-temporal techniques for TP in an efficient and accurate manner is an ongoing research problem. Existing TP approaches are limited to the short-term predictions. Moreover, they cannot handle a large volume of trajectory data for long-term prediction. To address these limitations, we propose a scalable clustering and Markov chain-based hybrid framework, called Traj-clusiVAT-based TP, for both short- and long-term TPs, which can handle a large number of overlapping trajectories in a dense road network. Traj-clusiVAT can also determine the number of clusters, which represent different movement behaviors in input trajectory data. In our experiments, we compare our proposed approach with a mixed Markov model-based scheme and a trajectory clustering, NETSCAN-based TP method for both short- and long-term TPs. We performed our experiments on two real, vehicle trajectory datasets, including a large-scale trajectory dataset consisting of 3.28 million trajectories obtained from 15 061 taxis in Singapore over a period of one month. The experimental results on two real trajectory datasets show that our proposed approach outperforms the existing approaches in terms of both short- and long-term prediction performances, based on the prediction accuracy and distance error (in km).

Clustering large volumes of high-dimensional data is a challenging task. Many clustering algorithms have been developed to address either handling datasets with a very large sample size or with a very high number of dimensions, but they are often impractical when the data is large in both aspects. To simultaneously overcome both the `curse of dimensionality' problem due to high dimensions and scalability problems due to large sample size, we propose a new fast clustering algorithm called FensiVAT. FensiVAT is a hybrid, ensemble-based clustering algorithm which uses fast data-space reduction and an intelligent sampling strategy. In addition to clustering, FensiVAT also provides visual evidence that is used to estimate the number of clusters (cluster tendency assessment) in the data. In our experiments, we compare FensiVAT with nine state-of-the-art approaches which are popular for large sample size or high-dimensional data clustering. Experimental results suggest that FensiVAT, which can cluster large volumes of high-dimensional datasets in a few seconds, is the fastest and most accurate method of the ones tested.

In this paper, we study the internal incremental Davies-Bouldin (iiDB) cluster validity index in the context of streaming data analysis. We extend the original index to a more general version parameterized by the exponent of membership weights. Then we illustrate how the iiDB can be used to analyze and understand the performance of the Extended Robust Online Streaming Clustering (EROLSC) algorithm. We give examples that illustrate the appearance of a new cluster, the effect of different cluster sizes, handling of outlier data samples, and the effect of the input order on the resultant cluster history.

The problem of determining whether clusters are present in numerical data (tendency assessment) is an important first step of cluster analysis. One tool for cluster tendency assessment is the visual assessment of tendency (VAT) algorithm. VAT and improved VAT (iVAT) produce an image that provides visual evidence about the number of clusters to seek in the original dataset. These methods have been successful in determining potential cluster structure in various datasets, but they can be computationally expensive for datasets with a very large number of samples. A scalable version of iVAT called siVAT approximates iVAT images, but siVAT can be computationally expensive for big datasets. In this article, we introduce a modification of siVAT called siVAT+ which approximates cluster heat maps for large volumes of high dimensional data much more rapidly than siVAT. We compare siVAT+ with siVAT on six large, high dimensional datasets. Experimental results confirm that siVAT+ obtains images similar to siVAT images in a few seconds, and is 8 - 55 times faster than siVAT.

Dunn's internal cluster validity index and its generalizations assess partition quality. For partitions of n samples of p-dimensional feature vector data, all but two of the generalized Dunn's indices (GDIs) have quadratic time complexity O(pn2), so computation is untenable for very large values of n. In this paper, we present two methods for approximating GDIs based on Maximin (MM) Sampling. MM sampling identifies a skeleton of the full partition that usually contains some of the boundary points in each cluster which are used to compute GDIs. We compare our algorithms with a support vector machine based boundary extraction method and a random sampling based estimation method. Our experiments on four real and synthetic datasets show that computing approximations to (three) GDIs with the MM skeleton is both computationally tractable and reliably accurate.

Inliers (bridge points) between clusters degrade the ability of many algorithms to find clusters in numerical data. We present three new approaches to the detection and removal of inliers. Two approaches are based on Local Outlier Factor (LOF) scores. We also discuss using LOF scores for an isolation Nearest Neighbour Ensemble (iNNE) approach to inlier detection. The third approach uses MaxiMin (MM) sampling to remove both inliers and outliers. We compare the three approaches on a synthetic and two real-life datasets. The failure of single linkage clustering due to the existence of bridging points is used as a means for evaluating the relative effectiveness of the three methods. We also show how inliers can degrade the quality of images built by the improved Visual Assessment of Tendency (iVAT) algorithm, which provides a visual representation of potential single linkage clusters in the data.

Presents a brief historical review of the various Editors in Chief of the IEEE Transactions on Fuzzy Systems.

Random projection is a popular method for dimensionality reduction due to its simplicity and efficiency. In the past few years, random projection and fuzzy c-means based cluster ensemble approaches have been developed for high-dimensional data clustering. However, they require large amounts of space for storing a big affinity matrix, and incur large computation time while clustering in this affinity matrix. In this paper, we propose a new random projection, fuzzy c-means based cluster ensemble framework for high-dimensional data. Our framework uses cumulative agreement to aggregate fuzzy partitions. Fuzzy partitions of random projections are ranked using external and internal cluster validity indices. The best partition in the ranked queue is the core (or base) partition. Remaining partitions then provide cumulative inputs to the core, thus, arriving at a consensus best overall partition built from the ensemble. Experimental results with Gaussian mixture datasets and a variety of real datasets demonstrate that our approach outperforms three state-of-the-art methods in terms of accuracy and space-time complexity. Our algorithm runs one to two orders of magnitude faster than other state-of-the-arts algorithms.

One-class support vector machines (OCSVMs) are very effective for semisupervised anomaly detection. However, their performance strongly depends on the settings of their hyperparameters, which has not been well studied. Moreover, unavailability of a clean training set that only comprises normal data in many real-life problems has given rise to the application of OCSVMs in an unsupervised manner. However, it has been shown that if the training set includes anomalies, the normal boundary created by OCSVMs is prone to skew toward the anomalies. This problem decreases the detection rate of anomalies and results in poor performance of the classifier. In this paper, we propose a new technique to set the hyperparameters and clean suspected anomalies from unlabelled training sets. The proposed method removes suspected anomalies based on a $K$ -nearest neighbors technique, which is then used to directly estimate the hyperparameters. We examine several benchmark data sets with diverse distributions and dimensionality. Our findings suggest that on the examined data sets, the proposed technique is roughly 70 times faster than supervised parameter estimation via grid-search and cross validation, and one to three orders of magnitude faster than broadly used semisupervised and unsupervised parameter estimation methods for OCSVMs. Moreover, our method statistically outperforms those semisupervised and unsupervised methods and its accuracy is comparable to supervised grid-search and cross validation.

The existence of large volumes of time series data in many applications has motivated data miners to investigate specialized methods for mining time series data. Clustering is a popular data mining method due to its powerful exploratory nature and its usefulness as a preprocessing step for other data mining techniques. This article develops two novel clustering algorithms for time series data that are extensions of a crisp c-shapes algorithm. The two new algorithms are heuristic derivatives of fuzzy c-means (FCM). Fuzzy c-Shapes plus (FCS+) replaces the inner product norm in the FCM model with a shape-based distance function. Fuzzy c-Shapes double plus (FCS++) uses the shape-based distance, and also replaces the FCM cluster centers with shape-extracted prototypes. Numerical experiments on 48 real time series data sets show that the two new algorithms outperform state-of-the-art shape-based clustering algorithms in terms of accuracy and efficiency. Four external cluster validity indices (the Rand index, Adjusted Rand Index, Variation of Information, and Normalized Mutual Information) are used to match candidate partitions generated by each of the studied algorithms. All four indices agree that for these finite waveform data sets, FCS++ gives a small improvement over FCS+, and in turn, FCS+ is better than the original crisp c-shapes method. Finally, we apply two tests of statistical significance to the three algorithms. The Wilcoxon and Friedman statistics both rank the three algorithms in exactly the same way as the four cluster validity indices.

Outlier detection is an important task in data mining. With the growing need to analyze high speed data streams, the task of outlier detection becomes even more challenging as traditional outlier detection techniques can no longer assume that all the data can be stored for processing. While the wellknown Local Outlier Factor (LOF) algorithm has an incremental version (called iLOF), it assumes unbounded memory to keep all previous data points. In this paper, we propose a memory efficient incremental local outlier (MiLOF) detection algorithm for data streams, and a more flexible version (MiLOF F), both have an accuracy close to iLOF but within a fixed memory bound. In addition MiLOF F is robust to changes in the number of data points, underlying clusters and dimensions in the data stream.

Efficient localized data modeling techniques in Internet of Things (IoT) applications enable the nodes to change their behavior upon observing events of interest. Additionally, battery-powered IoT nodes can conserve their energy resources by limiting their data communications to specific events. Despite the real-time nature of the data collected in the IoT and limited memory and computational resources, most of the current data modeling approaches for the IoT involve batch training. Recently, an online efficient anomaly detection technique called iterative data capture anomaly detection has been proposed for environmental sensing and monitoring applications. However, this approach cannot handle changing environments. So far, efforts in extending this algorithm to adapt to changes in the environment have met with limited success. In this paper, we generalize this algorithm to adapt to changes in the data stream by exponentially weighting past observations. We illustrate the proposed algorithm with numerical results on both real-life and simulated data sets, which demonstrate the efficiency and accuracy of our approach compared to existing methods.

Previously, eight popular information-theoretic-based cluster validity indices have been generalized and tested for probabilistic partitions built by the expectation-maximization (EM) algorithm for the Gaussian mixture model. However, the analysis was limited to probabilistic clusters, and there were limited explanations for differences in the performance of the indices. In this paper, we extend the tests to partitions found by fuzzy c-means (FCM) and provide further explanations and insights about the performance of these indices. Of the eight generalized indices, we advocate a normalized version of the soft mutual information cluster validity index (NMI sM) as the best overall choice, as it outperforms the other seven indices for both FCM and EM according to our tests on synthetic and real data. The superiority of NMIsM is most pronounced for datasets with overlapped and/or varying-sized clusters. Finally, we provide a theoretical analysis, which helps explain the superior performance of NMIsM compared with the other three normalizations of soft mutual information.

The Johnson-Lindenstrauss (JL) lemma, with known probability, sets a lower bound q0on the dimension for which a random projection of p-dimensional vector data is guaranteed to be within (1±ε) of being an isometry in a randomly projected downspace. We study several ways to identify a “good” rogue random projection when the target downspace has dimensions below the JL limit. The tools used towards this end are Pearson and Spearman correlation coefficients, and a visual imaging method (a cluster heat map) that usually reveals cluster structure in spaces of any dimension. We use four synthetic data sets and the ubiquitous Iris data to study our procedures for tracking the reliability of RRPs. Unsurprisingly, rogue random projection is quite unpredictable. At its best, it is every bit as good as Principal Components Analysis, but at it's worst, it is awful. Pearson and Spearman correlations do signal good and bad projections, but the visual imaging method seems even more effective in determining the quality of RRPs.

The intrinsic nature of streaming data requires algorithms that are capable of fast data analysis to extract knowledge. Most current unsupervised data analysis techniques rely on the implementation of known batch techniques over a sliding window, which can hinder their utility for the analysis of evolving structure in applications involving large streams of data. This research presents a novel data clustering algorithm, which exploits the correlation between data points in time to cluster the data, while maintaining a set of decision boundaries to identify noisy or anomalous data. We illustrate the proposed algorithm for online clustering with numerical results on both real-life and simulated datasets, which demonstrate the efficiency and accuracy of our approach compared to existing methods.

Clustering of big data has received much attention recently. In this paper, we present a new clusiVAT algorithm and compare it with four other popular data clustering algorithms. Three of the four comparison methods are based on the well known, classical batch k -means model. Specifically, we use k -means, single pass k -means, online k -means, and clustering using representatives (CURE) for numerical comparisons. clusiVAT is based on sampling the data, imaging the reordered distance matrix to estimate the number of clusters in the data visually, clustering the samples using a relative of single linkage (SL), and then noniteratively extending the labels to the rest of the data-set using the nearest prototype rule. Previous work has established that clusiVAT produces true SL clusters in compact-separated data. We have performed experiments to show that k -means and its modified algorithms suffer from initialization issues that cause many failures. On the other hand, clusiVAT needs no initialization, and almost always finds partitions that accurately match ground truth labels in labeled data. CURE also finds SL type partitions but is much slower than the other four algorithms. In our experiments, clusiVAT proves to be the fastest and most accurate of the five algorithms; e.g., it recovers 97% of the ground truth labels in the real world KDD-99 cup data (4 292 637 samples in 41 dimensions) in 76 s.

The C index is an internal cluster validity index that was introduced in 1970 as a way to define and identify a “best” crisp partition on n objects represented by either unlabeled feature vectors or dissimilarity matrix data. This index is often one of the better performers among the plethora of internal indices available for this task. This paper develops a soft generalization of the C index that can be used to evaluate sets of candidate partitions found by either fuzzy or probabilistic clustering algorithms. We define four generalizations based on relational transformations of the soft partition and, then, compare their performance to eight other popular internal fuzzy cluster indices using two methods of comparison (internal “best-c ” and internal/external (I/E) “best match”), six synthetic datasets, and six real-world labeled datasets. Our main conclusion is that the sum-min generalization is the second best performer in the best-c tests and the best performer in the I/E tests on small data.

The growth in pervasive network infrastructure called the Internet of Things (IoT) enables a wide range of physical objects and environments to be monitored in fine spatial and temporal detail. The detailed, dynamic data that are collected in large quantities from sensor devices provide the basis for a variety of applications. Automatic interpretation of these evolving large data is required for timely detection of interesting events. This article develops and exemplifies two new relatives of the visual assessment of tendency (VAT) and improved visual assessment of tendency (iVAT) models, which uses cluster heat maps to visualize structure in static datasets. One new model is initialized with a static VAT/iVAT image, and then incrementally (hence inc-VAT/inc-iVAT) updates the current minimal spanning tree (MST) used by VAT with an efficient edge insertion scheme. Similarly, dec-VAT/dec-iVAT efficiently removes a node from the current VAT MST. A sequence of inc-iVAT/dec-iVAT images can be used for (visual) anomaly detection in evolving data streams and for sliding window based cluster assessment for time series data. The method is illustrated with four real datasets (three of them being smart city IoT data). The evaluation demonstrates the algorithms’ ability to successfully isolate anomalies and visualize changing cluster structure in the streaming data.

The iVAT (asiVAT) algorithms reorder symmetric (asymmetric) dissimilarity data so that an image of the data may reveal cluster substructure. Images formed from incomplete data don't offer a very rich interpretation of cluster structure. In this paper, we examine four methods for completing the input data with imputed values before imaging. We choose a best method using contaminated versions of the complete Iris data, for which the desired results are known. Then, we analyze two real world data sets from social networks that are incomplete using the best imputation method chosen in the juried trials with Iris: (i) Sampson's monastery data, an incomplete, asymmetric relation matrix; and (ii) the karate club data, comprising a symmetric similarity matrix that is about 86 percent incomplete.

Outlier detection is an important task in data mining, with applications ranging from intrusion detection to human gait analysis. With the growing need to analyze high speed data streams, the task of outlier detection becomes even more challenging as traditional outlier detection techniques can no longer assume that all the data can be stored for processing. While the well-known Local Outlier Factor (LOF) algorithm has an incremental version, it assumes unbounded memory to keep all previous data points. In this paper, we propose a memory efficient incremental local outlier (MiLOF) detection algorithm for data streams, and a more flexible version (MiLOF_F), both have an accuracy close to Incremental LOF but within a fixed memory bound. Our experimental results show that both proposed approaches have better memory and time complexity than Incremental LOF while having comparable accuracy. In addition, we show that MiLOF_F is robust to changes in the number of data points, the number of underlying clusters and the number of dimensions in the data stream. These results show that MiLOF/MiLOF_F are well suited to application environments with limited memory (e.g., wireless sensor networks), and can be applied to high volume data streams.

This article was commissioned by the 22ndIEEE International Conference of Fuzzy Systems (FUZZ-IEEE) to celebrate the 50thAnniversary of Lotfi Zadeh's seminal 1965 paper on fuzzy sets. In addition to Lotfi's original paper, this note itemizes 100 citations of books and papers deemed “important (significant, seminal, etc.)” by 20 of the 21 living IEEE CIS Fuzzy Systems pioneers. Each of the 20 contributors supplied 5 citations, and Lotfi's paper makes the overall list a tidy 101, as in “Fuzzy Sets 101”. This note is not a survey in any real sense of the word, but the contributors did offer short remarks to indicate the reason for inclusion (e.g., historical, topical, seminal, etc.) of each citation. Citation statistics are easy to find and notoriously erroneous, so we refrain from reporting them - almost. The exception is that according to Google scholar on April 9, 2015, Lotfi's 1965 paper has been cited 55,479 times.

Many contemporary biomedical applications such as physiological monitoring, imaging, and sequencing produce large amounts of data that require new data processing and visualization algorithms. Algorithms such as principal component analysis (PCA), singular value decomposition and random projections (RP) have been proposed for dimensionality reduction. In this paper we propose a new random projection version of the fuzzy c-means (FCM) clustering algorithm denoted as RPFCM that has a different ensemble aggregation strategy than the one previously proposed, denoted as ensemble FCM (EFCM). RPFCM is more suitable than EFCM for big data sets (large number of points, n). We evaluate our method and compare it to EFCM on synthetic and real datasets.

Evolvable Takagi-Sugeno (T-S) models are fuzzy-rule-based models with the ability to continuously learn and adapt to incoming samples from data streams. The model adjusts both premise and consequent parameters to enhance the performance of the model. This paper introduces a new methodology for the estimation of the premise parameters in the evolvable T-S (eTS) model. Incremental updates for the weighted sample mean and inverse of the covariance matrix enable us to construct an evolvable fuzzy rule base that is used to detect outliers and regime changes in the input stream. We compare our model with Angelov's eTS+ model with artificial and real data.

We introduce a new method for multidimensional scaling in dissimilarity data that is based on preservation of metric topology between the original and derived data sets. The model seeks neighbors in the derived data that have the same ranks as in the input data. The algorithm we use to optimize the model is a modification of particle swarm optimization called multiswarming. We compare the new method to three well known approaches: Principal component analysis, Sammon's method, and (Kruskal's) metric MDS. Our method produces feature vector realizations that compare favorably with the other approaches on three real relational data sets.

There have been a large number of external validity indices proposed for cluster validity. One such class of cluster comparison indices is the information theoretic measures, due to their strong mathematical foundation and their ability to detect non-linear relationships. However, they are devised for evaluating crisp (hard) partitions. In this paper, we generalize eight information theoretic crisp indices to soft clusterings, so that they can be used with partitions of any type (i.e., crisp or soft, with soft including fuzzy, probabilistic and possibilistic cases). We present experimental results to demonstrate the effectiveness of the generalized information theoretic indices.

We present a model for the analysis of time series sensor data collected at an eldercare facility. The sensors measure restlessness in bed and bedroom motion of residents during the night. Our model builds sets of linguistic summaries from the sensor data that describe different events that may occur each night. A dissimilarity measure produces a distance matrix D between selected sets of summaries. Visual examination of the image of a reordered version of D provides an estimate for the number of clusters to seek in D. Then, clustering with single linkage or non-Euclidean relational fuzzy c-means produces groups of summaries. Subsequently, each group is represented by a linguistic medoid prototype. The prototypes can be used for resident monitoring, two types of anomaly detection, and interresident comparisons. We illustrate our model with real data for two residents collected at TigerPlace: the “aging in place” facility in Columbia, MO, USA.

Increased levels of particulate matter (PM) in the atmosphere have contributed to an increase in mortality and morbidity in communities and are the main contributing factor for respiratory health problems in the population. Currently, PM concentrations are sparsely monitored; for instance, a region of over 2200 square kilometers surrounding Melbourne in Victoria, Australia, is monitored using ten sensor stations. This paper proposes to improve the estimation of PM concentration by complementing the existing high-precision but expensive PM devices with low-cost lower precision PM sensor nodes. Our evaluation reveals that local PM estimation accuracies improve with higher densities of low-precision sensor nodes. Our analysis examines the impact of the precision of the lost-cost sensors on the overall estimation accuracy.

ABSTRACT Two new incremental models for online anomaly detection in data streams at nodes inwireless sensor networksare discussed. These models are incremental versions of a model that uses ellipsoids to detect first, second, and higher‐ordered anomaliesin arrears. The incremental versions can also be used this way but have additional capabilities offered by processing data incrementally as they arrive in time. Specifically, they can detect anomalies ‘on‐the‐fly’ in near real time. They can also be used to track temporal changes in near real‐time because of sensor drift, cyclic variation, or seasonal changes. One of the new models has a mechanism that enables graceful degradation of inputs in the distant past (fading memory). Three real datasets from single sensors in deployed environmental monitoring networks are used to illustrate various facets of the new models. Examples compare the incremental version with the previous batch and dynamic models and show that the incremental versions can detect various types of dynamic anomalies in near real time. Copyright © 2012 John Wiley & Sons, Ltd.

Recent algorithmic and computational improvements have reduced the time it takes to build a minimal spanning tree (MST) for big data sets. In this paper we compare single linkage clustering based on MSTs built with the Filter-Kruskal method to the proposed clusiVAT algorithm, which is based on sampling the data, imaging the sample to estimate the number of clusters, followed by non-iterative extension of the labels to the rest of the big data with the nearest prototype rule. Numerical experiments with both synthetic and real data confirm the theory that clusiVAT produces true single linkage clusters in compact, separated data. We also show that single linkage fails, while clusiVAT finds high quality partitions that match ground truth labels very well. And clusiVAT is fast: it recovers the preferred c = 3 Gaussian clusters in a mixture of 1 million two-dimensional data points with 100% accuracy in 3.1 seconds.

We discuss a new formulation of a fuzzy validity index that generalizes the Newman-Girvan (NG) modularity function. The NG function serves as a cluster validity functional in community detection studies. The input data is an undirected graph G = (V, E) that represents a social network. Clusters in V correspond to socially similar substructures in the network. We compare our fuzzy modularity to an existing modularity function using the well-studied Karate Club data set.

The iVAT algorithm reorders (symmetric) dissimilarity data so that an image of the data may reveal cluster substructure. This paper extends the method so that it can handle asymmetric dissimilarity data. The extension is based on replacing the asymmetric input data with its unique least-squared error approximation by a symmetric matrix. Examples are given to illustrate the new method, called asymmetric iVAT (asiVAT).

We introduce a new method for feature extraction from object data that is based on the idea of preserving metric topology between the original and derived data sets. Specifically, our method attempts to produce neighbors in the derived data that have the same ranks as in the input data. The algorithm we propose is a novel modification of particle swarm optimization that involves multiswarms. We compare our model and algorithm to feature extraction using Sammon's method and principal components analysis on 19 data sets: 17 are created by making draws from p-variate Gaussian distributions. We also use two real world data sets - the Glass and Lung Cancer data available at the UCI ML website. We find that the new method compares well with Sammon's method, and seems to be superior to features derived with principal components analysis.

We describe a new cluster validity framework (CVF) that compares structure in the data (in dissimilarity form) to the structure of dissimilarity matrices induced by a matrix transformation of the partition being tested. As part of this framework, we show two possible cluster validation measures: one, visual cluster validity, that that uses visual comparison and another one, correlation cluster validity, based on correlation. Unlike many existing measures, the measures we propose can be applied to crisp or soft partitions obtained by any relational or object data clustering algorithm. We illustrate the new measures and compare them to several well-known existing measures using real and artificial data sets.

We discuss a new formulation of a fuzzy validity index that generalizes the Newman-Girvan (NG) modularity function. The NG function serves as a cluster validity functional in community detection studies. The input data is an undirected weighted graph that represents, e.g., a social network. Clusters correspond to socially similar substructures in the network. We compare our fuzzy modularity with two existing modularity functions using the well-studied Karate Club and American College Football datasets.

This paper presents cluster validity for kernel fuzzy clustering. First, we describe existing cluster validity indices that can be directly applied to partitions obtained by kernel fuzzy clustering algorithms. Second, we show how validity indices that take dissimilarity (or relational) data D as input can be applied to kernel fuzzy clustering. Third, we present four propositions that allow other existing cluster validity indices to be adapted to kernel fuzzy partitions. As an example of how these propositions are used, five well-known indices are formulated.We demonstrate several indices for kernel fuzzy c-means (kFCM) partitions of both synthetic and real data.

Fuzzy c-means (FCM) is a well-known algorithm for clustering data, but for large datasets termination takes significant time. As a result, a number of scalable algorithms based on FCM have been developed. In this paper, four scalable variants of FCM are compared to the base algorithm. Runtime and three quality metrics are calculated. Experimental results using five data sets are analyzed. We show that the scalable algorithms are consistent with regard to speedup, but less consistent when quality is considered. The three quality measures are shown to have little correlation and vary in magnitude across datasets. Selection of a scalable algorithm must consider a tradeoff between the quality of results and speed. Of the variants, single pass FCM (SPFCM) is fastest with good fidelity to FCM, and extensible fast FCM (eFFCM) is almost as fast as SPFCM (as implemented) with very good fidelity to FCM. Random FCM is the fastest overall and often close in quality to FCM. The results showed that scalable algorithms occasionally produce better optimized results than FCM.

Linguistic summarization of time series can glean meaningful information from huge amounts of data. However, in situations like continuous monitoring, even linguistic summaries become difficult for a person to understand. In this paper, we develop an approach to generate linguistic prototypes from a group of time blocks that represent a normal condition. Then the set of summaries for new time blocks are compared to the prototypes to flag anomalous conditions, thereby reducing the burden on the human. Case studies from an eldercare environment demonstrate the utility of this approach.

One of the applications that motivates this research is a system for detection of the anomalies in wireless sensor networks (WSNs). Individual sensor measurements are converted to ellipsoidal summaries; a data matrix D is built using a dissimilarity measure between pairs of ellipsoids; clusters of ellipsoids are suggested by dark blocks along the diagonal of an iVAT (improved Visual Assessment of Tendency) image of D; and finally, the single linkage algorithm extracts clusters from D, using the iVAT image as a guide to the selection of an optimal partition. We illustrate this model for higher dimensional data with synthetic, real and benchmark data sets. Our examples show that two of the four measures, viz, Focal distance and Bhattacharyya distance, provide very similar and reliable bases for estimating cluster structures in sets of higher dimensional ellipsoids, that single linkage can successfully extract the indicated clusters, and that our model can find both first and second order anomalies in WSN data.

Since 1998, a graphical representation used in visual clustering called the reordered dissimilarity image or cluster heat map has appeared in more than 4000 biological or biomedical publications. These images are typically used to visually estimate the number of clusters in a data set, which is the most important input to most clustering algorithms, including the popularly chosen fuzzy c-means and crisp k-means. This paper presents a new formulation of a matrix reordering algorithm, coVAT, which is the only known method for providing visual clustering information on all four types of cluster structure in rectangular relational data. Finite rectangular relational data are an m× n array R of relational values between m row objects Or and n column objects Oc. R presents four clustering problems: clusters in Or, Oc, Or∪c, and coclusters containing some objects from each of Or and Oc. coVAT1 is a clustering tendency algorithm that provides visual estimates of the number of clusters to seek in each of these problems by displaying reordered dissimilarity images. We provide several examples where coVAT1 fails to do its job. These examples justify the introduction of coVAT2, a modification of coVAT1 based on a different reordering scheme. We offer several examples to illustrate that coVAT2 may detect coclusters in R when coVAT1 does not. Furthermore, coVAT2 is not limited to just relational data R. The R matrix can also take the form of feature data, such as gene microarray data where each data element is a real number: Positive values indicate upregulation, and negative values indicate downregulation. We show examples of coVAT2 on microarray data that indicate coVAT2 shows cluster tendency in these data. © 2012 Wiley Periodicals, Inc.

Very large (VL) data or big data are any data that you cannot load into your computer's working memory. This is not an objective definition, but a definition that is easy to understand and one that is practical, because there is a dataset too big for any computer you might use; hence, this is VL data for you. Clustering is one of the primary tasks used in the pattern recognition and data mining communities to search VL databases (including VL images) in various applications, and so, clustering algorithms that scale well to VL data are important and useful. This paper compares the efficacy of three different implementations of techniques aimed to extend fuzzy c-means (FCM) clustering to VL data. Specifically, we compare methods that are based on 1) sampling followed by noniterative extension; 2) incremental techniques that make one sequential pass through subsets of the data; and 3) kernelized versions of FCM that provide approximations based on sampling, including three proposed algorithms. We use both loadable and VL datasets to conduct the numerical experiments that facilitate comparisons based on time and space complexity, speed, quality of approximations to batch FCM (for loadable data), and assessment of matches between partitions and ground truth. Empirical results show that random sampling plus extension FCM, bit-reduced FCM, and approximate kernel FCM are good choices to approximate FCM for VL data. We conclude by demonstrating the VL algorithms on a dataset with 5 billion objects and presenting a set of recommendations regarding the use of different VL FCM clustering schemes.

The VAT algorithm is a visual method for determining the possible number of clusters in, or the cluster tendency of a set of objects. The improved VAT (iVAT) algorithm uses a graph-theoretic distance transform to improve the effectiveness of the VAT algorithm for “tough” cases where VAT fails to accurately show the cluster tendency. In this paper, we present an efficient formulation of the iVAT algorithm which reduces the computational complexity of the iVAT algorithm from O(N^3) to O(N^2). We also prove a direct relationship between the VAT image and the iVAT image produced by our efficient formulation. We conclude with three examples displaying clustering tendencies in three of the Karypis data sets that illustrate the improvement offered by the iVAT transformation. We also provide a comparison of iVAT images to those produced by the Reverse Cuthill-Mckee (RCM) algorithm; our examples suggest that iVAT is superior to the RCM method of display.

Previously, we presented a method for comparing soft partitions (i.e. crisp, probabilistic, fuzzy and possibilistic) to a known crisp reference partition. Many of the classical indices that have been used with outputs of crisp clustering algorithms were generalized so that they are applicable for candidate partitions of any type. In particular, focus was placed on generalizations of the Rand index. In this article, we extend our prior work by (1) investigating the behavior of the soft Rand for comparing non-crisp, specifically possibilistic, partitions and (2) we demonstrate how the possibilistic Rand and visual assessment of (cluster) tendency (VAT) algorithm can be used to discover the number of actual clusters and coincident clusters for outputs from the possibilistic c-means (PCM) algorithm.

Wireless Sensor Networks (WSNs) provide a low cost option for gathering spatially dense data from different environments. However, WSNs have limited energy resources that hinder the dissemination of the raw data over the network to a central location. This has stimulated research into efficient data mining approaches, which can exploit the restricted computational capabilities of the sensors to model their normal behavior. Having a normal model of the network, sensors can then forward anomalous measurements to the base station. Most of the current data modeling approaches proposed for WSNs require a fixed offline training period and use batch training in contrast to the real streaming nature of data in these networks. In addition they usually work in stationary environments. In this paper we present an efficient online model construction algorithm that captures the normal behavior of the system. Our model is capable of tracking changes in the data distribution in the monitored environment. We illustrate the proposed algorithm with numerical results on both real-life and simulated data sets, which demonstrate the efficiency and accuracy of our approach compared to existing methods.

A common question asked about unlabeled data sets is how many subsets (or clusters) of objects are represented in the data? The answer to this question is usually obtained by first clustering the data, and then employing a cluster validity measure to validate one or more candidate partitions of the objects. In this paper we describe an universal cluster validity measure that, unlike most existing measures, can be applied to partitions obtained by any relational or object data clustering algorithm. We illustrate the new measure, and compare it to several well known existing measures using a variety of artificial data sets.

We model anomalies in wireless sensor networks with ellipsoids that represent node measurements. Elliptical anomalies (EAs) are level sets of ellipsoids, and classify them as type 1, type 2 and higher order anomalies. Three measures of (dis)similarity between pairs of ellipsoids convert model ellipsoids into dissimilarity data. Clusters in the dissimilarity data may correspond to normal and anomalous measurements and nodes in the network. Assessment of (clustering) tendency is facilitated by visual inspection of (VAT/iVAT) images. Two examples illustrate the potential for anomaly detection.

The hard, fuzzy and possibilistic c-means clustering algorithms are widely used for partitioning a set of n objects into c groups. There are cases, however, when more than one type of partition is necessary to correctly describe the belongingness of an object to a group. Previously, Pal, Pal and Bezdek listed some of these cases and proposed a method to simultaneously produce both memberships and typicalities for a set of vectorial object data: the fuzzy possibilistic c-means (FPCM) clustering algorithm. However, FPCM is not directly applicable when the data are represented by object-object relationships. In this paper, we reformulate FPCM so that it can work with A-norm relational data. Extensions and properties of the relational clustering algorithm are also considered.

When clustering produces more than one candidate to partition a finite set of objectsO, there are two approaches to validation (i.e., selection of a “best” partition, and implicitly, a best value for c , which is the number of clusters inO). First, we may use an internal index, which evaluates each partition separately. Second, we may compare pairs of candidates with each other, or with a reference partition that purports to represent the “true” cluster structure in the objects. This paper generalizes many of the classical indices that have been used with outputs of crisp clustering algorithms so that they are applicable for candidate partitions of any type (i.e., crisp or soft, with soft comprising the fuzzy, probabilistic, and possibilistic cases). Space prevents inclusion of all of the possible generalizations that can be realized this way. Here, we concentrate on the Rand index and its modifications. We compare our fuzzy-Rand index with those of Campello, Hullermeier and Rifqi, and Brouwer, and show that our extension of the Rand index is O(n), while the other three are all O(n2). Numerical examples are given to illustrate various facets of the new indices. In particular, we show that our indices can be used, even when the partitions are probabilistic or possibilistic, and that our method of generalization is valid for any index that depends only on the entries of the classical (i.e., four-pair types) contingency table for this problem.

Numerous computational schemes have arisen over the years that attempt to learn information about objects based upon the similarity or dissimilarity of one object to another. One such scheme, clustering, looks for self-similar groups of objects. To use clustering algorithms, an investigator must often havea prioriknowledge of the number of clusters, i.e.,$c$, to search for in the data. Moreover, it is often convenient to have ways to rank the returned results, either for a single value of$c$, a range of$c$’s different clustering methods, or any combination thereof. However, the task of assessing the quality of the results, so that$c$may be determined objectively, is currently ill-defined for object–object relationships. To bridge this gap, we generalize three well-known validity indices: the modified Hubert’s Gamma, Xie–Beni, and the generalized Dunn’s indices, to relational data. In doing so, we develop a framework to convert many other validity indices to a relational form. Numerical examples on 12 datasets (samples from four normal mixtures, four real-world object datasets, and four real-world “pure relational” datasets) using the relational duals of the hard, fuzzy, and possibilistic$c$-means cluster algorithms are offered to illustrate and evaluate the new indices.

Clustering aims to identify groups of similar objects. To evaluate the results of cluster algorithms, an investigator uses cluster-validity indices. While the theory of cluster validity is well established for vector object data, little effort has been made to extend it to relationship-based data. As such, this paper proposes a theory of reformulation for object-data validity indices so that they can be used to rank the results produced by the relational -means clustering algorithms. More specifically, we create a class of relational validity indices, which is called dual-relational indices, that are guaranteed under certain, but easily met, constraints to produce the same results and, hence, the same cluster counts, as their object-data counterparts.

Anomaly detection in wireless sensor networks is an important challenge for tasks such as intrusion detection and monitoring applications. This paper proposes two approaches to detecting anomalies from measurements from sensor networks. The first approach is a linear programming-based hyperellipsoidal formulation, which is called a centered hyperellipsoidal support vector machine (CESVM). While this CESVM approach has advantages in terms of its flexibility in the selection of parameters and the computational complexity, it has limited scope for distributed implementation in sensor networks. In our second approach, we propose a distributed anomaly detection algorithm for sensor networks using a one-class quarter-sphere support vector machine (QSSVM). Here a hypersphere is found that captures normal data vectors in a higher dimensional space for each sensor node. Then summary information about the hyperspheres is communicated among the nodes to arrive at a global hypersphere, which is used by the sensors to identify any anomalies in their measurements. We show that the CESVM and QSSVM formulations can both achieve high detection accuracies on a variety of real and synthetic data sets. Our evaluation of the distributed algorithm using QSSVM reveals that it detects anomalies with comparable accuracy and less communication overhead than a centralized approach.

Visual methods have been widely studied and used in data cluster analysis. Given a pairwise dissimilarity matrix {\schmi D} of a set of n objects, visual methods such as the VAT algorithm generally represent {\schmi D} as an n\times n image {\rm I}(\tilde{{\schmi D}}) where the objects are reordered to reveal hidden cluster structure as dark blocks along the diagonal of the image. A major limitation of such methods is their inability to highlight cluster structure when {\schmi D} contains highly complex clusters. This paper addresses this limitation by proposing a Spectral VAT algorithm, where {\schmi D} is mapped to {\schmi D}^{\prime } in a graph embedding space and then reordered to {{\tilde{\schmi D}^{\prime }}} using the VAT algorithm. A strategy for automatic determination of the number of clusters in {\rm I}({\tilde{{\schmi D}^{\prime }}}) is then proposed, as well as a visual method for cluster formation from {\rm I}({\tilde{{\schmi D}^{\prime }}}) based on the difference between diagonal blocks and off-diagonal blocks. A sampling-based extended scheme is also proposed to enable visual cluster analysis for large data sets. Extensive experimental results on several synthetic and real-world data sets validate our algorithms.

The existence of BLAST sequence comparison algorithm and microarray technology are among the reasons that make bioinformatics the domain with the most abundant large relational datasets. For example, by BLAST-ing the genes of the human genome (around 30,000 genes) we obtain a 30,000 by 30,000 distance matrix. This matrix can not be currently stored in the memory of a typical desktop PC. In the same time, clustering the resulting matrix using a fuzzy relational clustering algorithm such as Non-Euclidean Fuzzy C-means (NERFCM) requires prior knowledge of the number of clusters existent in the data set. The question is, how can we evaluate the number of clusters if we can't even load the matrix in the memory our PC? To address this problem, we propose to extend the correlation cluster validity (CCV) that we introduced in a previous paper, denoting the new validity measure as eCCV. eCCV consists of two steps: first sampling of the large matrix followed by the estimation of the number of cluster employing CCV of the sampled data. The sampling strategy produces also a significant processing speedup. We illustrate eCCV properties on a large synthetic dataset and on a large subset of human genes obtained from the RefSeq database.

This paper presents a new technique for clustering either object or relational data. First, the data are represented as a matrix D of dissimilarity values. D is reordered to D* using a visual assessment of cluster tendency algorithm. If the data contain clusters, they are suggested by visually apparent dark squares arrayed along the main diagonal of an image I(D*) of D*. The suggested clusters in the object set underlying the reordered relational data are found by defining an objective function that recognizes this blocky structure in the reordered data. The objective function is optimized when the boundaries in I(D*) are matched by those in an aligned partition of the objects. The objective function combines measures of contrast and edginess and is optimized by particle swarm optimization. We prove that the set of aligned partitions is exponentially smaller than the set of partitions that needs to be searched if clusters are sought in D. Six numerical examples are given to illustrate various facets of the algorithm. © 2009 Wiley Periodicals, Inc.

Clustering is a popular tool for exploratory data analysis. One of the major problems in cluster analysis is the determination of the number of clusters in unlabeled data, which is a basic input for most clustering algorithms. In this paper we investigate a new method called DBE (dark block extraction) for automatically estimating the number of clusters in unlabeled data sets, which is based on an existing algorithm for visual assessment of cluster tendency (VAT) of a data set, using several common image and signal processing techniques. Basic steps include: 1) generating a VAT image of an input dissimilarity matrix; 2) performing image segmentation on the VAT image to obtain a binary image, followed by directional morphological filtering; 3) applying a distance transform to the filtered binary image and projecting the pixel values onto the main diagonal axis of the image to form a projection signal; 4) smoothing the projection signal, computing its first-order derivative, and then detecting major peaks and valleys in the resulting signal to decide the number of clusters. Our new DBE method is nearly "automatic", depending on just one easy-to-set parameter. Several numerical and real-world examples are presented to illustrate the effectiveness of DBE.

Anomalies in wireless sensor networks can occur due to malicious attacks, faulty sensors, changes in the observed external phenomena, or errors in communication. Defining and detecting these interesting events in energy-constrained situations is an important task in managing these types of networks. A key challenge is how to detect anomalies with few false alarms while preserving the limited energy in the network. In this article, we define different types of anomalies that occur in wireless sensor networks and provide formal models for them. We illustrate the model using statistical parameters on a dataset gathered from a real wireless sensor network deployment at the Intel Berkeley Research Laboratory. Our experiments with a novel distributed anomaly detection algorithm show that it can detect elliptical anomalies with exactly the same accuracy as that of a centralized scheme, while achieving a significant reduction in energy consumption in the network. Finally, we demonstrate that our model compares favorably to four other well-known schemes on four datasets.

This paper presents an ontological self-organizing map (OSOM), which is used to produce visualization and functional summarization information about gene products using gene ontology (GO) similarity measures. The OSOM is an extension of the self-organizing map as initially developed by Kohonen, which trains on data composed of sets of terms. Term-based similarity measures are used as a distance metric as well as in the update of the OSOM training procedure. We present an OSOM-based visualization method that shows the cluster tendency of the gene products. Also demonstrated is an OSOM-based functional summarization which produces the most representative term(s) (MRT) from the GO for each OSOM prototype and, subsequently, each gene product cluster. We validated the results of our method by applying the OSOM to a well-studied set of gene products.

Many important applications in biology have underlying datasets that are relational, that is, only the (dis)similarity between biological objects (amino acid sequences, gene expression profiles, etc.) is known and not their feature values in some feature space. Examples of such relational datasets are the gene similarity matrices obtained from BLAST, gene expression data, or gene ontology (GO) similarity measures. Once a relational dataset is obtained, a common question asked is how many groups of objects are represented in the original dataset. The answer to this question is usually obtained by employing a clustering algorithm and a cluster validity measure. In this article we describe a cluster validity measure for non-Euclidean relational fuzzy c-means that is based on the correlation between a relation induced on the data by the cluster memberships and the original relational data. This validity measure can be applied to partitions made by any fuzzy relational clustering algorithm. We illustrate our measure by validating clusters in several dissimilarity matrices for a set of 194 gene products obtained using BLAST and GO similarities.

Given a pairwise dissimilarity matrix D of a set of objects, visual methods such as the VAT algorithm (for visual analysis of cluster tendency) represent (D macr )as an image (D macr ) where the objects are reordered to highlight cluster structure as dark blocks along the diagonal of the image. A major limitation of such visual methods is their inability to highlight cluster structure in 1(D macr ) when D contains clusters with highly complex structure. In this paper, we address this limitation by proposing a Spectral VAT (SpecVAT) algorithm, where D is mapped to D' in an embedding space by spectral decomposition of the Laplacian matrix, and then reordered to D' using the VAT algorithm. We also propose a strategy to automatically determine the number of clusters in (D macr '), as well as a method for cluster formation from (D macr ') based on the difference between diagonal blocks and off-diagonal blocks. We demonstrate the effectiveness of our algorithms on several synthetic and real-world data sets that are not amenable to analysis via traditional VAT.

This paper addresses the relationship between the visual assessment of cluster tendency (VAT) algorithm and Dunnpsilas cluster validity index. We present an analytical comparison in conjunction with numerical examples to demonstrate that the effectiveness of VAT in showing cluster tendency is directly related to Dunnpsilas index. This analysis is important to understanding the underlying theory of VAT and VAT-based algorithms and, more generally, other algorithms that are based on, or similar to, Primpsilas Algorithm.

A key challenge in pattern recognition is how to scale the computational efficiency of clustering algorithms on large data sets. The extension of non-Euclidean relational fuzzy c-means (NERF) clustering to very large (VL = unloadable) relational data is called the extended NERF (eNERF) clustering algorithm, which comprises four phases: (i) finding distinguished features that monitor progressive sampling; (ii) progressively sampling from a N × N relational matrix RN to obtain a n × n sample matrix Rn; (iii) clustering Rn with literal NERF; and (iv) extending the clusters in Rn to the remainder of the relational data. Previously published examples on several fairly small data sets suggest that eNERF is feasible for truly large data sets. However, it seems that phases (i) and (ii), i.e., finding Rn, are not very practical because the sample size n often turns out to be roughly 50% of n, and this over-sampling defeats the whole purpose of eNERF. In this paper, we examine the performance of the sampling scheme of eNERF with respect to different parameters. We propose a modified sampling scheme for use with eNERF that combines simple random sampling with (parts of) the sampling procedures used by eNERF and a related algorithm sVAT (scalable visual assessment of clustering tendency). We demonstrate that our modified sampling scheme can eliminate over-sampling of the original progressive sampling scheme, thus enabling the processing of truly VL data. Numerical experiments on a distance matrix of a set of 3,000,000 vectors drawn from a mixture of 5 bivariate normal distributions demonstrate the feasibility and effectiveness of the proposed sampling method. We also find that actually running eNERF on a data set of this size is very costly in terms of computation time. Thus, our results demonstrate that further modification of eNERF, especially the extension stage, will be needed before it is truly practical for VL data. © 2008 Wiley Periodicals, Inc.

Anomaly detection is an important challenge for tasks such as fault diagnosis and intrusion detection in energy constrained wireless sensor networks. A key problem is how to minimise the communication overhead in the network while performing in-network computation when detecting anomalies. Our approach to this problem is based on a formulation that uses distributed, one-class quarter-sphere support vector machines to identify anomalous measurements in the data. We demonstrate using sensor data from the Great Duck Island Project that our distributed approach is energy efficient in terms of communication overhead while achieving comparable accuracy to a centralised scheme.

We have an m times n matrix D, and assume that its entries correspond to pair wise dissimilarities between m row objects Orand n column objects Oc, which, taken together (as a union), comprise a set O of N = m + n objects. This paper develops a new visual approach that applies to four different cluster assessment problems associated with O. The problems are the assessment of cluster tendency: PI) amongst the row objects Or; P2) amongst the column objects Oc; P3) amongst the union of the row and column objects OrU Oc; and P4) amongst the union of the row and column objects that contain at least one object of each type (co-clusters). The basis of the method is to regard D as a subset of known values that is part of a larger, unknown N times N dissimilarity matrix, and then impute the missing values from D. This results in estimates for three square matrices (Dr, Dc, DrUc) that can be visually assessed for clustering tendency using the previous VAT or sVAT algorithms. The output from assessment of DrUcultimately leads to a rectangular coVAT image which exhibits clustering tendencies in D. Five examples are given to illustrate the new method. Two important points: i) because VAT is scalable by sVAT to data sets of arbitrary size, and because coVAT depends explicitly (and only) on VAT, this new approach is immediately scalable to, say, the scoVAT model, which works for even very large (unloadable) data sets without alteration; and ii) VAT, sVAT and coVAT are autonomous, parameter free models - no "hidden values" are needed to make them work.

Different extensions of fuzzy c-means (FCM) clustering have been developed to approximate FCM clustering in very large (unloadable) image (eFFCM) and object vector (geFFCM) data. Both extensions share three phases: (1) progressive sampling of the VL data, terminated when a sample passes a statistical goodness of fit test; (2) clustering with (literal or exact) FCM; and (3) noniterative extension of the literal clusters to the remainder of the data set. This article presents a comparable method for the remaining case of interest, namely, clustering in VL relational data. We will propose and discuss each of the four phases of eNERF and our algorithm for this last case: (1) finding distinguished features that monitor progressive sampling, (2) progressively sampling a square N × N relation matrix RN until an n × n sample relation Rn passes a statistical test, (3) clustering Rn with literal non-Euclidean relational fuzzy c-means, and (4) extending the clusters in Rn to the remainder of the relational data. The extension phase in this third case is not as straightforward as it was in the image and object data cases, but our numerical examples suggest that eNERF has the same approximation qualities that eFFCM and geFFCM do. © 2006 Wiley Periodicals, Inc. Int J Int Syst 21: 817–841, 2006.

The standard method for comparing gene products (proteins or RNA) is to compare their DNA or amino acid sequences. Additional information about some gene products may come from multiple sources, including the set of Gene Ontology (GO) annotations and the set of journal abstracts related to each gene product. Gene product similarity measures can be based on evaluating sets of descriptor terms found in the GO taxonomy, and/or the index term sets of the related documents (MeSH annotations). While our techniques can be applied to term sets from any taxonomy, we restrict our examples in this article to GO annotations. We investigate the use of linear order statistics (LOS) to build similarity relations on pairs of terms that are used in the GO as linguistic descriptors of genes and gene products. One of our objectives is to investigate the construction and utility of visual assessments of relational data (in this case, dissimilarity matrices) for discovering tendencies of groups of gene products to "cluster together". We use gene product data derived from a group of 194 gene products representing three protein families extracted from ENSEMBL. Our examples suggest that LOS similarity measures are more effective than traditional sequence-based similarity measures at capturing relationships between pairs of gene products in ENSEMBL families when annotation information is available. We show examples of how these similarity measures can assist in knowledge discovery and gene product family validation.

Successes with kernel-based classification methods have spawned recent efforts to kernelize clustering algorithms for object data. Here we extend the kernelization to relational data clustering by proposing a kernelized form of the nonEuclidean relational fuzzy c-means algorithm. A numerical test result is included

The mountain clustering method and the subtractive clustering method are useful methods for finding cluster centers based on local density in object data. These methods have been extended to shell clustering. In this article, we propose a relational mountain clustering method (RMCM), which produces a set of (proto) typical objects as well as a crisp partition of the objects generating the relation, using a new concept that we call relational density. We exemplify RMCM by clustering several relational data sets that come from object data. Finally, RMCM is applied to web log analysis, where it produces useful user profiles from web log data. © 2005 Wiley Periodicals, Inc. Int J Int Syst 20: 375–392, 2005.

In 1997, we proposed the fuzzy-possibilistic c-means (FPCM) model and algorithm that generated both membership and typicality values when clustering unlabeled data. FPCM constrains the typicality values so that the sum over all data points of typicalities to a cluster is one. The row sum constraint produces unrealistic typicality values for large data sets. In this paper, we propose a new model called possibilistic-fuzzy c-means (PFCM) model. PFCM produces memberships and possibilities simultaneously, along with the usual point prototypes or cluster centers for each cluster. PFCM is a hybridization of possibilistic c-means (PCM) and fuzzy c-means (FCM) that often avoids various problems of PCM, FCM and FPCM. PFCM solves the noise sensitivity defect of FCM, overcomes the coincident clusters problem of PCM and eliminates the row sum constraints of FPCM. We derive the first-order necessary conditions for extrema of the PFCM objective function, and use them as the basis for a standard alternating optimization approach to finding local minima of the PFCM objective functional. Several numerical examples are given that compare FCM and PCM to PFCM. Our examples show that PFCM compares favorably to both of the previous models. Since PFCM prototypes are less sensitive to outliers and can avoid coincident clusters, PFCM is a strong candidate for fuzzy rule-based system identification.

The extensible fast fuzzy c-means algorithm (eFFCM) finds clusters in very large digital images. eFFCM identifies a representative subsample of the image, which is then clustered using the fuzzy c-means (FCM) algorithm. The subsample solution is then extended to secure an approximate clustering of the remaining pixels in the image. This article discusses generalized eFFCM (geFFCM), the extension of eFFCM to general non-image data. Our extension accelerates literal fuzzy c-means (LFCM) on all (loadable) data sets. Second, geFFCM provides feasibility - a way to find (approximate) clusters - for data sets that are too large to be loaded in a single computer. Our experiments suggest that the chi-squared or divergence test for goodness of fit alone identifies good subsamples. This new subsampling method should be equally effective for acceleration and feasibility with VL data by any extensible clustering algorithm (not just FCM).

Earlier we proposed the fuzzy-possibilistic c-means (FPCM) model and algorithm that generated both membership and typicality values when clustering unlabeled data. FPCM imposes a constraint on the sum of typicalities over a cluster that leads to unrealistic typicality values for large data sets. Here we propose a new model called possibilistic fuzzy c-means (PFCM). PFCM produces memberships and possibilities simultaneously, along with the cluster centers. PFCM addresses the noise sensitivity defect of FCM, overcomes the coincident clusters problem of possibilistic c-means (PCM) and eliminates the row sum constraints of FPCM. Our numerical examples show that PFCM compares favorably to all of the previous models.

Conventional cluster validity techniques usually represent all the validity information available about a particular clustering by a single number. The display method introduced here is an alternative to standard validity functionals. The proposed approach uses intensity images generated from the results of any prototype generator clustering algorithm as a means for cluster validation. Several numerical examples are given to illustrate the method.

This paper presents an offline word-recognition system based on structural information in the unconstrained written word. Oriented features in the word are extracted with the Gabor filters. We estimate the Gabor filter parameters from the grayscale images. A two-dimensional fuzzy word classification system is developed where the spatial location and shape of the membership functions are derived from the training words. The system achieves an average recognition rate of 74% for the word being correctly classified in the top position and an average of 96% for the word being correctly classified within the top five positions.

We present a method for sampling feature vectors in large (e.g., 2000 /spl times/ 5000 /spl times/ 16 bit) images that finds subsets of pixel locations which represent c "regions" in the image. Samples are accepted by the chi-square (/spl chi//sup 2/) or divergence hypothesis test. A framework that captures the idea of efficient extension of image processing algorithms from the samples to the rest of the population is given. Computationally expensive (in time and/or space) image operators (e.g., neural networks (NNs) or clustering models) are trained on the sample, and then extended noniteratively to the rest of the population. We illustrate the general method using fuzzy c-means (FCM) clustering to segment Indian satellite images. On average, the new method can achieve about 99% accuracy (relative to running the literal algorithm) using roughly 24% of the image for training. This amounts to an average savings of 76% in CPU time. We also compare our method to its closest relative in the group of schemes used to accelerate FCM: our method averages a speedup of about 4.2, whereas the multistage random sampling approach achieves an average acceleration of 1.63.

We compare eleven methods for finding prototypes upon which to base the nearest prototype classifier. Four methods for prototype selection are discussed: Wilson+Hart (a condensation+error-editing method), and three types of combinatorial search—random search, genetic algorithm, and tabu search. Seven methods for prototype extraction are discussed: unsupervised vector quantization, supervised learning vector quantization (with and without training counters), decision surface mapping, a fuzzy version of vector quantization, c-means clustering, and bootstrap editing. These eleven methods can be usefully divided two other ways: by whether they employ pre- or postsupervision; and by whether the number of prototypes found is user-defined or “automatic.” Generalization error rates of the 11 methods are estimated on two synthetic and two real data sets. Offering the usual disclaimer that these are just a limited set of experiments, we feel confident in asserting that presupervised, extraction methods offer a better chance for success to the casual user than postsupervised, selection schemes. Finally, our calculations do not suggest that methods which find the “best” number of prototypes “automatically” are superior to methods for which the user simply specifies the number of prototypes. © 2001 John Wiley & Sons, Inc.

We discuss target detection in LADAR intensity images. Thirteen features, eleven of which come from an asymmetric co-occurrence matrix, are extracted from region-of-interest windows in each image. Two methods of feature selection are applied to the extracted vectors. Random selection leads to a pair of selected features for a nearest-neighbor rule (1-nn) detector. Extended backpropagation leads to six selected features using a modified multilayered perceptron (MLP) network. The 1-nn detector achieves a test-error rate of about 16% at a false-alarm rate of 8%. The MLP has a test-error rate of about 12% with a false-alarm rate of 6%.

The problem of clustering a real s-dimensional data set X={x(1 ),,,,,x(n)} subset R(s) is considered. Usually, each observation (or datum) consists of numerical values for all s features (such as height, length, etc.), but sometimes data sets can contain vectors that are missing one or more of the feature values. For example, a particular datum x(k) might be incomplete, having the form x(k)=(254.3, ?, 333.2, 47.45, ?)(T), where the second and fifth feature values are missing. The fuzzy c-means (FCM) algorithm is a useful tool for clustering real s-dimensional data, but it is not directly applicable to the case of incomplete data. Four strategies for doing FCM clustering of incomplete data sets are given, three of which involve modified versions of the FCM algorithm. Numerical convergence properties of the new algorithms are discussed, and all approaches are tested using real and artificially generated incomplete data sets.

In this paper, we present an image understanding system using fuzzy sets and fuzzy measures. This system is based on a symbolic object-oriented image interpretation system. We apply a simple, powerful three-dimensional (3-D) recursive filter to tracking moving objects in a dynamic image sequence. This filter has a time-varying 3-D frequency-planar passband that is adapted in a feedback system to automatically track moving objects. However, as objects in the image sequence are not well-defined and are engaged in dynamic activities, their shapes and trajectories in most cases can be described only vaguely. In order to handle these uncertainties, we use fuzzy measures to capture subtle variations and manage the uncertainties involved. This enables us to develop an image understanding system that produces a very natural output. We demonstrate the effectiveness of our system with complex real traffic scenes.

We describe the use of segmentation with fuzzy models and classification by the crisp k-nearest neighbor (k-nn) algorithm for assisting breast cancer detection in digital mammograms. Our research utilizes images from the digital database for screening mammography. We show that supervised and unsupervised methods of segmentation, such as k-nn and fuzzy c-means, in digital mammograms will have high misclassification rates when only intensity is used as the discriminating feature. Adding window means and standard deviations to the feature suite (visually) improves segmentation produced by the k-nn rule. While our results are encouraging, other methods are needed to detect smaller pathologies such as microcalcifications.

Alternating cluster estimation (ACE) is a generalized clustering model. Relational ACE is a modification of ACE that can be used to cluster data which do not possess a clear numerical representation, but for which a meaningful relation matrix can be defined. For text data sets we define (pairwise) relation matrices based on the Levenshtein string distance (1966). Relational ACE with Levenshtein distances is applied to four different texts. The cluster centers represent typical words in the texts, so this algorithm can be used to automatically determine keywords.

Fuzzy c-means (FCM) is a useful clustering technique. Modifications of FCM using L/sub 1/ norm distances increase robustness to outliers. Object and relational data versions of FCM clustering are defined for the more general case where the L/sub p/ norm (p/spl ges/1) or semi-norm (00 in order to facilitate the empirical examination of the object data models. Both object and relational approaches are included in a numerical study.

Describes a genetically guided approach to optimizing the hard (J/sub 1/) and fuzzy (J/sub m/) c-means functionals used in cluster analysis. Our experiments show that a genetic algorithm (GA) can ameliorate the difficulty of choosing an initialization for the c-means clustering algorithms. Experiments use six data sets, including the Iris data, magnetic resonance, and color images. The genetic algorithm approach is generally able to find the lowest known J/sub m/ value or a J/sub m/ associated with a partition very similar to that associated with the lowest J/sub m/ value. On data sets with several local extrema, the GA approach always avoids the less desirable solutions. Degenerate partitions are always avoided by the GA approach, which provides an effective method for optimizing clustering models whose objective function can be represented in terms of cluster centers. A series random initializations of fuzzy/hard c-means, where the partition associated with the lowest J/sub m/ value is chosen, can produce an equivalent solution to the genetic guided clustering approach given the same amount of processor time in some domains.

This correspondence points out several published errors in replicates of the well-known iris data, which was collected by Anderson (1935) but first published by Fisher (1936).

Many clustering models define good clusters as extrema of objective functions. Optimization of these models is often done using an alternating optimization (AO) algorithm driven by necessary conditions for local extrema. We abandon the objective function model in favor of a generalized model called alternating cluster estimation (ACE). ACE uses an alternating iteration architecture, but membership and prototype functions are selected directly by the user. Virtually every clustering model can be realized as an instance of ACE. Out of a large variety of possible instances of non-AO models, we present two examples: 1) an algorithm with a dynamically changing prototype function that extracts representative data and 2) a computationally efficient algorithm with hyperconic membership functions that allows easy extraction of membership functions. We illustrate these non-AO instances on three problems: a) simple clustering of plane data where we show that creating an unmatched ACE algorithm overcomes some problems of fuzzy c-means (FCM-AO) and possibilistic c-means (PCM-AO); b) functional approximation by clustering on a simple artificial data set; and c) functional approximation on a 12 input 1 output real world data set. ACE models work pretty well in all three cases.

We extend the nearest prototype classifier to a generalized nearest prototype classifier (GNPC). The GNPC uses "soft" labeling of the prototypes in the classes, thereby encompassing a variety of classifiers. Based on how the prototypes are found we distinguish between presupervised and postsupervised GNPC designs. We derive the conditions for optimality (relative to the standard Bayes error rate) of two designs where prototypes represent: 1) the components of class-conditional mixture densities (presupervised design) or 2) the components of the unconditional mixture density (postsupervised design). An artificial data set and the "satimage" data set from the database ELENA are used to experimentally study the two approaches. A radial basis function (RBF) network is used as a representative of each GNPC type. Neither the theoretical nor the experimental results indicate clear reasons to prefer one of the approaches. The postsupervised GNPC design tends to be more robust and less accurate than the presupervised one.

This paper describes a class of models we call semi-supervised clustering. Algorithms in this category are clustering methods that use information possessed by labeled training data Xd⊂ ℜp as well as structural information that resides in the unlabeled data Xu⊂ ℜp. The labels are used in conjunction with the unlabeled data to help clustering algorithms partition Xu ⊂ ℜp which then terminate without the capability to label other points in ℜp. This is very different from supervised learning, wherein the training data subsequently endow a classifier with the ability to label every point in ℜp. The methodology is applicable in domains such as image segmentation, where users may have a small set of labeled data, and can use it to semi-supervise classification of the remaining pixels in a single image. The model can be used with many different point prototype clustering algorithms. We illustrate how to attach it to a particular algorithm (fuzzy c-means). Then we give two numerical examples to show that it overcomes the failure of many point prototype clustering schemes when confronted with data that possess overlapping and/or non uniformly distributed clusters. Finally, the new method compares favorably to the fully supervised k nearest neighbor rule when applied to the Iris data.

We propose a Generalized Nearest Prototype Classifier (GNPC) as a common framework for a number of classification techniques. Specifically we consider clustering-and-relabeling; Parzen's classifier; radial basis functions (RBF) networks; learning vector quantization (LVQ) type classifiers; and nearest neighbor rules. To classify an unlabeled point x the GNPC combines the degrees of similarity of x to a set of prototypes. Five questions are addressed for these GNPC families: (1) How many prototypes do we need? (2) How are the prototypes found? (3) How are their class labels obtained? (4) How are the similarities defined? (5) How are the similarities and label information combined? The classification performance of a set of GNPCs is illustrated on two benchmark data sets: IRIS and the 2-spirals data. We study the resubstitution error of the GNPC as a function of the number of prototypes. Our conclusions are that: (a) unsupervised selection (or extraction) of prototypes followed by relabeling is inferior to the techniques that use labels to guide them towards prototypes; (b) the edited nearest neighbor rule is a viable option for GNPC design which has not received the attention it deserves.

This paper describes edge detection as a composition of four steps: conditioning, feature extraction, blending, and scaling. We examine the role of geometry in determining good features for edge detection and in setting parameters for functions to blend the features. We find that: (1) statistical features such as the range and standard deviation of window intensities can be as effective as more traditional features such as estimates of digital gradients; (2) blending functions that are roughly concave near the origin of feature space ran provide visually better edge images than traditional choices such as the city-block and Euclidean norms; (3) geometric considerations ran be used to specify the parameters of generalized logistic functions and Takagi-Sugeno input-output systems that yield a rich variety of edge images; and (4) understanding the geometry of the feature extraction and blending functions is the key to using models based on computational learning algorithms such as neural networks and fuzzy systems for edge detection. Edge images derived from a digitized mammogram are given to illustrate various facets of our approach.

Two models are discussed that integrate heterogeneous fuzzy data of three types: real numbers, real intervals, and real fuzzy sets. The architecture comprises three modules: 1) an encoder that converts the mixed data into a uniform internal representation; 2) a numerical processing core that uses the internal representation to solve a specified task; and 3) a decoder that transforms the internal representation back to an interpretable output format. The core used in this study is fuzzy clustering, but there are many other operations that are facilitated by the models. Two schemes for encoding the data and decoding it after clustering are presented. One method uses possibility and necessity measures for encoding and several variants of a center of gravity defuzzification method for decoding. The second approach uses piecewise linear splines to encode the data and decode the clustering results. Both procedures are illustrated using two small sets of heterogeneous fuzzy data.

We review two clustering algorithms (hard c-means and single linkage) and three indexes of crisp cluster validity (Hubert's statistics, the Davies-Bouldin index, and Dunn's index). We illustrate two deficiencies of Dunn's index which make it overly sensitive to noisy clusters and propose several generalizations of it that are not as brittle to outliers in the clusters. Our numerical examples show that the standard measure of interset distance (the minimum distance between points in a pair of sets) is the worst (least reliable) measure upon which to base cluster validation indexes when the clusters are expected to form volumetric clouds. Experimental results also suggest that intercluster separation plays a more important role in cluster validation than cluster diameter. Our simulations show that while Dunn's original index has operational flaws, the concept it embodies provides a rich paradigm for validation of partitions that have cloud-like clusters. Five of our generalized Dunn's indexes provide the best validation results for the simulations presented.

Five methods that generate multiple prototypes from labeled data are reviewed. Then we introduce a new sixth approach, which is a modification of Chang's (1974) method. We compare the six methods with two standard classifier designs: the 1-nearest prototype (1-np) and 1-nearest neighbor (1-nn) rules. The standard of comparison is the resubstitution error rate; the data used are the Iris data. Our modified Chang's method produces the best consistent (zero-error) design. One of the competitive learning models produces the best minimal prototypes design (five prototypes that yield three resubstitution errors).

Three questions related to the nearest prototype classifier (NPC) are addressed: when is it better to construct the prototypes instead of selecting them as a subset of the given labeled data; how can we trade classification accuracy for a reduction in the number of prototypes; and how good is pure random search (RS) for selection of prototypes from the data? We compare the resubstitution performance of the NPC on the IRIS data set, where the prototypes are either extracted by replacement (R-prototypes) or by selection (S-prototypes). Results for the R-prototypes are taken from a previous study and are contrasted with S-prototype results obtained by a genetic algorithm (GA) or by RS. The best results reached by both algorithms (GA and RS), followed by resubstitution NPC, are two errors with sets of three S-prototypes. This compares favorably to the best result found with R-prototypes, viz., three errors with five R-prototypes. Based on our results, we recommend GA selection for the NPC. A by-product of this research is a counter example to minimality of a recently published minimal consistent set selection procedure.

Derives an interpretation for a family of competitive learning algorithms and investigates their relationship to fuzzy c-means and fuzzy learning vector quantization. These algorithms map a set of feature vectors into a set of prototypes associated with a competitive network that performs unsupervised learning. Derivation of the new algorithms is accomplished by minimizing an average generalized distance between the feature vectors and prototypes using gradient descent. A close relationship between the resulting algorithms and fuzzy c-means is revealed by investigating the functionals involved. It is also shown that the fuzzy c-means and fuzzy learning vector quantization algorithms are related to the proposed algorithms if the learning rate at each iteration is selected to satisfy a certain condition.

Validation of partitions produced by the fuzzy c-means (FCM) clustering algorithm was discussed by Pal and Bezdek [1] . Two tables in [1] contain erroneous values. This errata reports the correct values, and notes that the conclusions drawn in [1], based on these simulations, remain unchanged.

When clustering algorithms are applied to image segmentation, the goal is to solve a classification problem. However, these algorithms do not directly optimize classification duality. As a result, they are susceptible to two problems: 1) the criterion they optimize may not be a good estimator of "true" classification quality, and 2) they often admit many (suboptimal) solutions. This paper introduces an algorithm that uses cluster validity to mitigate problems 1 and 2. The validity-guided (re)clustering (VGC) algorithm uses cluster-validity information to guide a fuzzy (re)clustering process toward better solutions. It starts with a partition generated by a soft or fuzzy clustering algorithm. Then it iteratively alters the partition by applying (novel) split-and-merge operations to the clusters. Partition modifications that result in improved partition validity are retained. VGC is tested on both synthetic and real-world data. For magnetic resonance image (MRI) segmentation, evaluations by radiologists show that VGC outperforms the (unsupervised) fuzzy c-means algorithm, and VGC's performance approaches that of the (supervised) k-nearest-neighbors algorithm.

Presented is a model that integrates three data types (numbers, intervals, and linguistic assessments). Data of these three types come from a variety of sensors. One objective of sensor-fusion models is to provide a common framework for data integration, processing, and interpretation. That is what our model does. We use a small set of artificial data to illustrate how problems as diverse as feature analysis, clustering, cluster validity, and prototype classifier design can all be formulated and attacked with standard methods once the data are converted to the generalized coordinates of our model. The effects of reparameterization on computational outputs are discussed. Numerical examples illustrate that the proposed model affords a natural way to approach problems which involve mixed data types.

First, we identify an algorithmic defect of the generalized learning vector quantization (GLVQ) scheme that causes it to behave erratically for a certain scaling of the input data. We show that GLVQ can behave incorrectly because its learning rates are reciprocally dependent on the sum of squares of distances from an input vector to the node weight vectors. Finally, we propose a new family of models-the GLVQ-F family-that remedies the problem. We derive competitive learning algorithms for each member of the GLVQ-F model and prove that they are invariant to all scalings of the data. We show that GLVQ-F offers a wide range of learning models since it reduces to LVQ as its weighting exponent (a parameter of the algorithm) approaches one from above. As this parameter increases, GLVQ-F then transitions to a model in which either all nodes may be excited according to their (inverse) distances from an input or in which the winner is excited while losers are penalized. And as this parameter increases without limit, GLVQ-F updates all nodes equally. We illustrate the failure of GLVQ and success of GLVQ-F with the IRIS data.

Various hard, fuzzy and possibilistic clustering criteria (objective functions) are useful as bases for a variety of pattern recognition problems. At present, many of these criteria have customized individual optimization algorithms. Because of the specialized nature of these algorithms, experimentation with new and existing criteria can be very inconvenient and costly in terms of development and implementation time. This paper shows how to reformulate some clustering criteria so that specialized algorithms can be replaced by general optimization routines found in commercially available software. We prove that the original and reformulated versions of each criterion are fully equivalent. Finally, two numerical examples are given to illustrate reformulation.>

Many functionals have been proposed for validation of partitions of object data produced by the fuzzy c-means (FCM) clustering algorithm. We examine the role a subtle but important parameter-the weighting exponent m of the FCM model-plays in determining the validity of FCM partitions. The functionals considered are the partition coefficient and entropy indexes of Bezdek, the Xie-Beni (1991), and extended Xie-Beni indexes, and the Fukuyama-Sugeno index (1989). Limit analysis indicates, and numerical experiments confirm, that the Fukuyama-Sugeno index is sensitive to both high and low values of m and may be unreliable because of this. Of the indexes tested, the Xie-Beni index provided the best response over a wide range of choices for the number of clusters, (2-10), and for m from 1.01-7. Finally, our calculations suggest that the best choice for m is probably in the interval [1.5, 2.5], whose mean and midpoint, m=2, have often been the preferred choice for many users of FCM.>

This paper discusses Kohonen's self-organizing semantic map (SOSM). We show that augmentation and normalization of numerical feature data as recommended for the SOSM is entirely unnecessary to obtain semantic maps that exhibit semantic similarities between objects represented by the data. Visual displays of a small data set of 13 animals based on principal components, Sammon's algorithm, and Kohonen's (unsupervised) self-organizing feature map (SOFM) possess exactly the same qualitative information as the much more complicated SOSM display does.

Replying to the comments of W.H. Woodall and R.E. Davis (ibid., p.43) on the author's editorial (ibid., p.1), the authors present two illustrations to show that they do not have a poor opinion of probability but that they believe fuzzy membership to be more useful in some cases.>

First, this paper reviews several well known measures of fuzziness for discrete fuzzy sets. Then new multiplicative and additive classes are defined. We show that each class satisfies five well-known axioms for fuzziness measures, and demonstrate that several existing measures are relatives of these classes. The multiplicative class is based on nonnegative, monotone increasing concave functions. The additive class requires only nonnegative concave functions. Some relationships between several existing and the new measures are established, and some new properties are derived. The relative merits and drawbacks of different measures for applications are discussed. A weighted fuzzy entropy which is flexible enough to incorporate subjectiveness in the measure of fuzziness is also introduced. Finally, we comment on the construction of measures that may assess all of the uncertainties associated with a physical system.>

IT is my great pleasure to welcome you to the inaugural issue of the IEEE TRANSACTIONOSN FUZZY SYSTEMS. Many of you are probably quite familiar with the basic ideas underlying fuzzy sets and systems that utilize fuzzy models. However, there will also be readers who are looking at the contents of this issue, wondering what it's all about. For this latter group, the papers in this first issue may seem bewildering, for they are quite technical, and none are tutorial in nature. Consequently, this preface is divided into two parts. Section I contains a brief introduction to the basic ideas of our science, in hopes that this glimpse will help you understand the articles, but perhaps more importantly, pique your interest in fuzzy models. Section I1 gives a short prCcis and historical perspective of the field, and how it has led to the creation of this journal.

A family of objective functions called fuzzy c-regression models, which can be used too fit switching regression models to certain types of mixed data, is presented. Minimization of particular objective functions in the family yields simultaneous estimates for the parameters of c regression models, together with a fuzzy c-partitioning of the data. A general optimization approach for the family of objective functions is given and corresponding theoretical convergence results are discussed. The approach is illustrated by two numerical examples that show how it can be used to fit mixed data to coupled linear and nonlinear models.>

The relationship between the sequential hard c-means (SHCM) and learning vector quantization (LVQ) clustering algorithms is discussed. The impact and interaction of these two families of methods with Kohonen's self-organizing feature mapping (SOFM), which is not a clustering method but often lends ideas to clustering algorithms, are considered. A generalization of LVQ that updates all nodes for a given input vector is proposed. The network attempts to find a minimum of a well-defined objective function. The learning rules depend on the degree of distance match to the winner node; the lesser the degree of match with the winner, the greater the impact on nonwinner nodes. Numerical results indicate that the terminal prototypes generated by this modification of LVQ are generally insensitive to initialization and independent of any choice of learning coefficient. IRIS data obtained by E. Anderson's (1939) is used to illustrate the proposed method. Results are compared with the standard LVQ approach.

R. N. Dave's (1990) version of fuzzy c-shells is an iterative clustering algorithm which requires the application of Newton's method or a similar general optimization technique at each half step in any sequence of iterates for minimizing the associated objective function. An important computational question concerns the accuracy of the solution required at each half step within the overall iteration. The general convergence theory for grouped coordination minimization is applied to this question to show that numerically exact solution of the half-step subproblems in Dave's algorithm is not necessary. One iteration of Newton's method in each coordinate minimization half step yields a sequence obtained using the fuzzy c-shells algorithm with numerically exact coordinate minimization at each half step. It is shown that fuzzy c-shells generates hyperspherical prototypes to the clusters it finds for certain special cases of the measure of dissimilarity used.

Magnetic resonance (MR) brain section images are segmented and then synthetically colored to give visual representations of the original data with three approaches: the literal and approximate fuzzy c-means unsupervised clustering algorithms, and a supervised computational neural network. Initial clinical results are presented on normal volunteers and selected patients with brain tumors surrounded by edema. Supervised and unsupervised segmentation techniques provide broadly similar results. Unsupervised fuzzy algorithms were visually observed to show better segmentation when compared with raw image data for volunteer studies. For a more complex segmentation problem with tumor/edema or cerebrospinal fluid boundary, where the tissues have similar MR relaxation behavior, inconsistency in rating among experts was observed, with fuzz-c-means approaches being slightly preferred over feedforward cascade correlation results. Various facets of both approaches, such as supervised versus unsupervised learning, time complexity, and utility for the diagnostic process, are compared.

An extension of the hard and fuzzy c-means (HCM/FCM) clustering algorithms is described. Specifically, these models are extended to admit the case where the (dis)similarity measure on pairs of numerical vectors includes two members of the Minkowski or p-norm family, viz., the p=1 and p= infinity norms. In the absence of theoretically necessary conditions to guide a numerical solution of the nonlinear constrained optimization problem associated with this case, it is shown that a certain basis exchange algorithm can be used to find approximate critical points of the new objective functions. This method broadens the applications horizon of the FCM family by enabling users to match discontinuous multidimensional numerical data structures with similarity measures that have nonhyperelliptical topologies.>

This paper discusses the design and implementation of PLAYMAKER, a knowledge-based system for characterizing hydrocarbon plays. PLAYMAKER is a component of XX (eXpert eXplorer), a workstation-based tool that aids exploration geologists in a number of different tasks: sediment and carbonate simulation, play and field characterization, retrieval and storage of information in a geological database, comparison of the play or field under study with other fields in the database, and report generation. PLAYMAKER is implemented using MIDST (Mixed Inferencing Dempster-Shafer Tool), a rule-based expert system shell that incorporates mixed-initiative and inexact reasoning based on the Dempster-Shafer evidence combination scheme. This paper discusses the effectiveness of a two-level knowledge base structure adopted for the design and implementation of PLAYMAKER.

The authors present Pool2, a generic system for cognitive map development and decision analysis that is based on negative-positive-neutral (NPN) logics and NPN relations. NPN logics and relations are extensions of two-valued crisp logic, crisp (binary) relations, and fuzzy relations, NPN logics and relations assume logic values in the NPN interval (-1, 1) instead of values in (0, 1). A theorem is presented that provides conditions for the existence and uniqueness of heuristic transitive closures of an NPN relation. It is shown that NPN logic and NPN relations can be used directly to model a target world with a combination of NPN relationships of attributes and/or concepts for the purposes of cognitive map understanding, and decision analysis. Two algorithms are presented for heuristic transitive closure computation and for heuristic path searching, respectively. Basic ideas are illustrated by example. A comparison is made between this approach and others.>

In this article we describe the conceptual model and processing of (constrained) natural-language queries in information retrieval systems. A language interface based on fuzzy set techniques is proposed to handle the uncertainty inherent in natural-language semantics. The conceptual model is developed and exemplified in the context of document retrieval. Specifically, the user query is considered to be a triple, q = (qc, qy, qn), where qc indicates the part of the query that deals with concepts and operators that link these concepts, qy identifies the publication period the user is interested in, and qn, pertains to the number of documents to be retrieved. We describe query decomposition using an augmented transition network parser and the assignment of functions and relations needed by each portion of the query to represent uncertainties inherent in the natural language. The output of the natural-language interface is then passed to a knowledge-based retrieval mechanism that will be described in a companion article (Part II). © 1987 John Wiley & Sons, Inc.

This article presents our conceptual model of the retrieval process of a document-retrieval system. The retrieval mechanism input is an unambiguous intermediate form of a user query generated by the language processor using the method described previously. Our retrieval mechanism uses a two-step procedure. In the first step a list of documents pertinent to the query are obtained from the document database, and then an evidence-combination scheme is used to compute the degree of support between the query and individual documents. The second step uses a ranking procedure to obtain a final degree of support for each document chosen, as a function of individual degrees of support associated with one or more parts of the query. The end result is a set of document citations presented to the user in ranked order in response to the information request. Numerical examples are given to illustrate various facets of the overall system, which has been prototypically implemented in modular form to test system response to changes in model parameters. © 1987 John Wiley & Sons, Inc.

A knowledge-based system approach is applied to the design of a document retrieval system for online retrieval of bibliographic material. The main components of the system are a language interface that understands user queries in pseudo natural language (i.e. queries that are restricted to subject domain concepts), a retrieval component that combines the expertise of subject-domain and retrieval experts, a help system that aids users in formulating and reformulating queries, and a user model builder that infers specific user characteristics. A comprehensive mathematical model based on fuzzy sets and fuzzy relations has been defined for the thesaurus and the retrieval mechanism and a prototypical system has been implemented in a modular fashion to test system response to changes in model parameters.

This paper reports the results of a numerical comparison of two versions of the fuzzy c-means (FCM) clustering algorithms. In particular, we propose and exemplify an approximate fuzzy c-means (AFCM) implementation based upon replacing the necessary ``exact'' variates in the FCM equation with integer-valued or real-valued estimates. This approximation enables AFCM to exploit a lookup table approach for computing Euclidean distances and for exponentiation. The net effect of the proposed implementation is that CPU time during each iteration is reduced to approximately one sixth of the time required for a literal implementation of the algorithm, while apparently preserving the overall quality of terminal clusters produced. The two implementations are tested numerically on a nine-band digital image, and a pseudocode subroutine is given for the convenience of applications-oriented readers. Our results suggest that AFCM may be used to accelerate FCM processing whenever the feature space is comprised of tuples having a finite number of integer-valued coordinates.

A low-level segmentation methodology based upon fuzzy clustering principles is developed. The approach utilizes region growing concepts and a pyramid data structure for the hierarchical analysis of aerial images. It is assumed that measurement vectors corresponding to perceptually homogeneous regions cluster together in the measurement space. The fuzzy c-means (FCM) clustering algorithm is used in the formulation. Utilization of the fuzzy partitioning allows one to derive a correspondence between the cluster membership function values and (the proportions of) the classes constituting a region. Thus cluster membership values can be used to split mixture regions into smaller regions at a higher resolution level. The feasibility of the methodology is evaluated using a three-channel Landsat image. The results show that the FCM clustering can be used in the single-level segmentation; and that cluster membership function values derived using this algorithm can be utilized effectively as indicators of region homogeneity.

An algorithm is described that fits boundary data of planar shapes in either rectangular coordinate or chain-code format with a set of straight line segments. The algorithm combines a new vertex detection method, which locates initial vertices and segments in the data, with thec-elliptotype clustering algorithm, which iteratively adjusts the location of these initial segments, thereby obtaining a best polygonal fit for the data in the mean-squared error sense. Several numerical examples are given to exemplify the implementation and utility of this new approach.

This paper introduces a new theory for the tangential deflection and curvature of plane discrete curves. Our theory applies to discrete data in either rectangular boundary coordinate or chain coded formats: its rationale is drawn from the statistical and geometric properties associated with the eigenvalue-eigenvector structure of sample covariance matrices. Specifically, we prove that the nonzero entry of the commutator of a piar of scatter matrices constructed from discrete arcs is related to the angle between their eigenspaces. And further, we show that this entry is-in certain limiting cases-also proportional to the analytical curvature of the plane curve from which the discrete data are drawn. These results lend a sound theoretical basis to the notions of discrete curvature and tangential deflection; and moreover, they provide a means for computationally efficient implementation of algorithms which use these ideas in various image processing contexts. As a concrete example, we develop the commutator vertex detection (CVD) algorithm, which identifies the location of vertices in shape data based on excessive cummulative tangential deflection; and we compare its performance to several well established corner detectors that utilize the alternative strategy of finding (approximate) curvature extrema.

In this paper the convergence of a class of clustering procedures, popularly known as the fuzzy ISODATA algorithms, is established. The theory of Zangwill is used to prove that arbitrary sequences generated by these (Picard iteration) procedures always terminates at a local minimum, or at worst, always contains a subsequence which converges to a local minimum of the generalized least squares objective functional which defines the problem.

The fuzzy ISODATA algorithms are used to address two problems: first, the question of feature selection for binary valued data sets is investigated; and second, the same method is applied to the design of a fuzzy one-nearest prototype classifier. The efficiency of this fuzzy classifier is compared to conventional k-NN classifiers by a computational example using the stomach disease data of Scheinok and Rupe, and Toussaint's method for estimation of the probability of misclassification: the fuzzy prototype classifier appears to decrease the error rate expected from all k-NN classifiers by roughly ten per cent.

An algorithm is described for generating fuzzy partitions which extremize a fuzzy extension of the k-means squared-error criterion function on finite data sets X. It is shown how this algorithm may be applied to the problem of estimating the parameters (a priori probabilities, means, and covariances) of mixture of multivariate normal densities, given a finite sample X drawn from the mixture. The behavior of the algorithm is compared with that of the ordinary ISODATA clustering process and the maximum likelihood method, for a specific bivariate mixture.