Auditability and verifiability are critical elements in establishing trustworthiness in federated learning (FL). These principles promote transparency, accountability, and independent validation of FL processes. Incorporating auditability and verifiability is imperative for building trust and ensuring the robustness of FL methodologies. Typical FL architectures rely on a trustworthy central authority to manage the FL process. However, reliance on a central authority could become a single point of failure, making it an attractive target for cyber-attacks and insider frauds. Moreover, the central entity lacks auditability and verifiability, which undermines the privacy and security that FL aims to ensure. This article proposes an auditable and verifiable decentralized FL (DFL) framework. We first develop a smart-contract-based monitoring system for DFL participants. This monitoring system is then deployed to each DFL participant and executed when the local model training is initiated. The monitoring system records necessary information during the local training process for auditing purposes. Afterward, each DFL participant sends the local model and monitoring system to the respective blockchain node. The blockchain nodes representing each DFL participant exchange the local models and use the monitoring system to validate each local model. To ensure an auditable and verifiable decentralized aggregation procedure, we record the aggregation steps taken by each blockchain node in the aggregation contract. Following the aggregation phase, each blockchain node applies a multisignature scheme to the aggregated model, producing a globally verifiable model. Based on the signed global model and the aggregation contract, each blockchain node implements a consensus protocol to store the validated global model in tamper-proof storage. To evaluate the performance of our proposed model, we conducted a series of experiments with different machine learning architectures and datasets, including CIFAR-10, F-MNIST, and MedMNIST. The experimental results indicate a slight increase in time consumption compared with the state-of-the-art, serving as a tradeoff to ensure auditability and verifiability. The proposed blockchain-enabled DFL also saves up to 95% communication costs for the participant side.

In contrast to traditional adversarial learning (AL) which learns speaker-invariant representations, this paper proposes cascaded adversarial learning (CAL) which learns speaker-invariant emotion data for speaker independent emotion recognition (SIER) tasks. CAL is a dual cascaded network architecture where the output of the transformation network is fed as input to the classification network. Transformation network transforms original speech emotion to speaker-invariant emotion data by implementing an AL strategy with an encoder-decoder architecture. The classification network predicts the emotion from the speaker-invariant emotion data (output of the transformation network). We argue that the speaker-invariant emotion data realized by transformation network has less variation than the original speech emotion data and therefore are conducive for SIER as it improve generalization capability. To our knowledge this is the first time a dual cascaded network has been used for SIER and demonstrate state-of-the-art performances for SIER on Emo-DB and RAVDESS datasets.

Currently, illegal parking detection tasks are mainly achieved through manually checking by enforcement officers on patrol or using Closed-Circuit Television (CCTV) cameras. However, these methods either need high human labour costs or demand installation costs and procedures. Therefore, illegal parking detection solutions, which can reduce significant labour and equipment installation costs, are highly demanded. This paper proposes a novel voting based detection algorithm using deep learning networks implemented using in-vehicle cameras to achieve illegal parking detection with multiple offences’ types. Adopting in-vehicle cameras better matches real-world mobile scenarios than using traditional CCTV cameras as this helps enforcement authorities to reduce manpower and installation costs. A well-constructed new dataset with more than 10000 high-quality labelled images with seven object categories is built for illegal parking detection tasks. Additionally, one novel labelling method named “minimal illegal units” is proposed for illegal parking detection. It reduces the time and human labelling costs significantly, achieving a better correlation of a vehicle and its parking type. The experiments have been conducted in the urban areas of Singapore. Furthermore, the illumination robustness test has also been performed to illustrate that the proposed detection algorithm exhibits strong resistance to changing illumination conditions in varied operating environments. Our proposed detection algorithm can provide a benchmark for research in illegal parking detection.

Convolutional dictionary learning (CDL) aims to learn a structured and shift-invariant dictionary to decompose signals into sparse representations. While yielding superior results compared to traditional sparse coding methods on various signal and image processing tasks, most CDL methods have difficulties handling large data, because they have to process all images in the dataset in a single pass. Therefore, recent research has focused on online CDL (OCDL) which updates the dictionary with sequentially incoming signals. In this article, a novel OCDL algorithm is proposed based on a local, slice-based representation of sparse codes. Such representation has been found useful in batch CDL problems, where the convolutional sparse coding and dictionary learning problem could be handled in a local way similar to traditional sparse coding problems, but it has never been explored under online scenarios before. We show, in this article, that the proposed algorithm is a natural extension of the traditional patch-based online dictionary learning algorithm, and the dictionary is updated in a similar memory efficient way too. On the other hand, it can be viewed as an improvement of existing second-order OCDL algorithms. Theoretical analysis shows that our algorithm converges and has lower time complexity than existing counterpart that yields exactly the same output. Extensive experiments are performed on various benchmarking datasets, which show that our algorithm outperforms state-of-the-art batch and OCDL algorithms in terms of reconstruction objectives.

Due to the outbreak of the novel coronavirus (or known as COVID-19), people are advised to wear masks when they stay outdoors in many countries. This could result in difficulty for some public safety surveillance systems involving face detection or tracking. Therefore, the development of face detection and tracking algorithms for people wearing face masks is particularly important. In this paper, a real-time tracking algorithm for people with or without face masks is proposed. This algorithm is trained on public face datasets with faces without masks. Although the training does not involve face images of people wearing face masks, we show that the proposed algorithm is robust as it is able to perform well in face tracking for people wearing face masks. We also discuss the possible scenarios where the algorithm could lose track of the target when experimenting in tracking masked faces. This can motivate future research in this area.

Noise that afflicts natural images, regardless of the source, generally disturbs the perception of image quality by introducing a high-frequency random element that, when severe, can mask image content. Except at very low levels, where it may play a purpose, it is annoying. There exist significant statistical differences between distortion-free natural images and noisy images that become evident upon comparing the empirical probability distribution histograms of their discrete wavelet transform (DWT) coefficients. The DWT coefficients of low- or no-noise natural images have leptokurtic, peaky distributions with heavy tails; while noisy images tend to be platykurtic with less peaky distributions and shallower tails. The sample kurtosis is a natural measure of the peakedness and tail weight of the distributions of random variables. Here, we study the efficacy of the sample kurtosis of image wavelet coefficients as a feature driving, an extreme learning machine which learns to map kurtosis values into perceptual quality scores. The model is trained and tested on five types of noisy images, including additive white Gaussian noise, additive Gaussian color noise, impulse noise, masked noise, and high-frequency noise from the LIVE, CSIQ, TID2008, and TID2013 image quality databases. The experimental results show that the trained model has better quality evaluation performance on noisy images than existing blind noise assessment models, while also outperforming general-purpose blind and full-reference image quality assessment methods.

Inspection of metal surface textures using computer vision and machine learning techniques plays an important role in Automated Visual Inspection (AVI) systems. Texture recognition on metal surface is challenging because the characteristics of each texture type are dependent on the properties of the metal surface when captured under different lighting conditions. Since these textures have no obvious repetitive patterns like general textures, this results in high intra-class diversities. Prior knowledge has shown that surface properties such as surface curvature and depth are discriminant to different texture types on metal surface. Since scale, shapes and location of textures within the same type are not fixed, scale property and spatial ordering information are less important for differentiating between texture types. There-fore, surface property, scale invariance and order-less property should be considered when exploring a suitable image feature for metal surface texture recognition. This paper proposes Order-less Scale Invariant Gradient Local Auto-Correlation (OS-GLAC) which meets all three requirements for robust texture recognition. The experiment results show that OS-GLAC is robust to separate different metal surface texture types. In addition, we observed that OS-GLAC is not only useful for texture recognition on metal surface but also for general texture recognition when combined with pre-trained deep learning features as these two features capture complimentary information. The experiment results show that such a combination of OS-GLAC achieves competitive results on three well-established general texture datasets i.e., KTH-TIP-2a, KTH-TIPS-2b and FMD.

Electronic tongue (E-Tongue), as a novel taste analysis tool, shows a promising perspective for taste recognition. In this paper, we constructed a voltammetric E-Tongue system and measured 13 different kinds of liquid samples, such as tea, wine, beverage, functional materials, etc. Owing to the noise of system and a variety of environmental conditions, the acquired E-Tongue data shows inseparable patterns. To this end, from the viewpoint of algorithm, we propose a local discriminant preservation projection (LDPP) model, an under-studied subspace learning algorithm, that concerns the local discrimination and neighborhood structure preservation. In contrast with other conventional subspace projection methods, LDPP has two merits. On one hand, with local discrimination it has a higher tolerance to abnormal data or outliers. On the other hand, it can project the data to a more separable space with local structure preservation. Further, support vector machine, extreme learning machine (ELM), and kernelized ELM (KELM) have been used as classifiers for taste recognition in E-Tongue. Experimental results demonstrate that the proposed E-Tongue is effective for multiple tastes recognition in both efficiency and effectiveness. Particularly, the proposed LDPP-based KELM classifier model achieves the best taste recognition performance of 98%. The developed benchmark data sets and codes will be released and downloaded in http://www.leizhang.tk/ tempcode.html.

In many practical transfer learning scenarios, the feature distribution is different across the source and target domains (i.e., nonindependent identical distribution). Maximum mean discrepancy (MMD), as a domain discrepancy metric, has achieved promising performance in unsupervised domain adaptation (DA). We argue that the MMD-based DA methods ignore the data locality structure, which, up to some extent, would cause the negative transfer effect. The locality plays an important role in minimizing the nonlinear local domain discrepancy underlying the marginal distributions. For better exploiting the domain locality, a novel local generative discrepancy metric-based intermediate domain generation learning called Manifold Criterion guided Transfer Learning (MCTL) is proposed in this paper. The merits of the proposed MCTL are fourfold: 1) the concept of manifold criterion (MC) is first proposed as a measure validating the distribution matching across domains, and DA is achieved if the MC is satisfied; 2) the proposed MC can well guide the generation of the intermediate domain sharing similar distribution with the target domain, by minimizing the local domain discrepancy; 3) a global generative discrepancy metric is presented, such that both the global and local discrepancies can be effectively and positively reduced; and 4) a simplified version of MCTL called MCTL-S is presented under a perfect domain generation assumption for more generic learning scenario. Experiments on a number of benchmark visual transfer tasks demonstrate the superiority of the proposed MC guided generative transfer method, by comparing with the other state-of-the-art methods. The source code is available in https://github.com/wangshanshanCQU/MCTL.

Most of the existing image blurriness assessment algorithms are proposed based on measuring image edge width, gradient, high-frequency energy, or pixel intensity variation. However, these methods are content sensitive with little consideration of image content variations, which causes variant estimations for images with different contents but same blurriness degrees. In this paper, a content-insensitive blind image blurriness assessment metric is developed utilizing Weibull statistics. Inspired by the property that the statistics of image gradient magnitude (GM) follows Weibull distribution, we parameterize the GM using$\beta$(scale parameter) and$\gamma$(shape parameter) of Weibull distribution. We also adopt skewness ($\eta$) to measure the asymmetry of the GM distribution. In order to reduce the influence of image content and achieve more robust performance, divisive normalization is then incorporated to moderate the$\beta$,$\gamma$, and$\eta$. The final image quality is predicted using a sparse extreme learning machine. Performances evaluation on the blur image subsets in LIVE, CSIQ, TID2008, and TID2013 databases demonstrate that the proposed method is highly correlated with human perception and robust with image contents. In addition, our method has low computational complexity which is suitable for online applications.

Convolutional sparse coding (CSC) has become an important method in image processing and computer vision. In this paper we focus on visual recognition problems and apply CSC as a feature learning method. We propose a task-specific approach to treat the dictionary of CSC as parameters for a larger learning framework. These parameters are differentiable under mild conditions, and could be updated end-to-end using back-propagation when the errors from the task objectives are provided. We perform several experiments to show that such method provides a more discriminate representation compared with previous CSC methods, and this data driven approach is effective for visual recognition problems.

We introduce Octree-based Convolutional Autoencoder Extreme Learning Machine (OCA-ELM) for 3D shape classification. This approach combines Convolutional Autoencoder Extreme Learning Machine (CAE-ELM) with octreebased con- volution to generate feature maps from several types of geometric data, and extract discriminative features with Extreme Learning Machine Autoencoder (ELM-AE). The extracted features can then be used for various computer graphics applications, such as 3D shape classification. Compared with other 3D classification methods, the proposed OCA-ELM has superior classification performance. Experiments on ModelNet40 show that OCA-ELM outperforms state-of-the-art CNN-based methods and surpasses CAE-ELM in classification accuracy by 3.69%, demonstrating the effectiveness of our method.

The automatic identification system (AIS) tracks vessel movement by means of electronic exchange of navigation data between vessels, with onboard transceiver, terrestrial, and/or satellite base stations. The gathered data contain a wealth of information useful for maritime safety, security, and efficiency. Because of the close relationship between data and methodology in marine data mining and the importance of both of them in marine intelligence research, this paper surveys AIS data sources and relevant aspects of navigation in which such data are or could be exploited for safety of seafaring, namely traffic anomaly detection, route estimation, collision prediction, and path planning.

Feature learning is an initial step applied to computer vision tasks and is broadly categorized as: 1) deep feature learning; 2) shallow feature learning. In this paper we focus on shallow feature learning as these algorithms require less computational resources than deep feature learning algorithms. In this paper we propose a shallow feature learning algorithm referred to as Extreme Learning Machine Network (ELMNet). ELMNet is module based neural network consist of feature learning module and a post-processing module. Each feature learning module in ELMNet performs the following operations: 1) patch-based mean removal; 2) ELM auto-encoder (ELM-AE) to learn features. Post-processing module is inserted after the feature learning module and simplifies the features learn by the feature learning modules by hashing and block-wise histogram. Proposed ELMNet outperforms shallow feature learning algorithm PCANet on the MNIST handwritten dataset.

Fully connected multi layer neural networks such as Deep Boltzmann Machines (DBM) performs better than fully connected single layer neural networks in image classification tasks and has a smaller number of hidden layer neurons than Extreme Learning Machine (ELM) based fully connected multi layer neural networks such as Multi Layer ELM (MLELM) and Hierarchical ELM (H-ELM) However, ML-ELM and H-ELM has a smaller training time than DBM. This paper introduces a fully connected multi layer neural network referred to as Multi Layer Multi Objective Extreme Learning Machine (MLMO-ELM) which uses a multi objective formulation to pass the label and non-linear information in order to learn a network model which has a similar number of hidden layer parameters as DBM and smaller training time than DBM. The experimental results show that MLMO-ELM outperforms DBM, ML-ELM and H-ELM on OCR and NORB datasets.

Word embeddings are a low-dimensional vector representation of words that incorporates context. TWo popular methods are word2vec and global vectors (GloVe). Word2vec is a single-hidden layer feedforward neural network (SLFN) that has an auto-encoder influence for computing a word context matrix using backpropagation for training. GloVe computes the word context matrix first then performs matrix factorization on the matrix to arrive at word embeddings. Backpropagation is a typical training method for SLFN's, which is time consuming and requires iterative tuning. Extreme learning machines (ELM) have the universal approximation capability of SLFN's, based on a randomly generated hidden layer weight matrix in lieu of backpropagation. In this research, we propose an efficient method for generating word embeddings that uses an auto-encoder architecture based on ELM that works on a word context matrix. Word similarity is done using the cosine similarity measure on a dozen various words and the results are reported.

A biological neural network is constituted by numerous subnetworks and modules with different functionalities. For an artificial neural network, the relationship between a network and its subnetworks is also important and useful for both theoretical and algorithmic research, i.e. it can be exploited to develop incremental network training algorithm or parallel network training algorithm. In this paper we explore the relationship between an Extreme Learning Machine (ELM) trained neural network and its subnetworks. To the best of our knowledge, we are the first to prove a theorem that shows an ELM trained neural network can be scattered into subnetworks and its optimal solution can be constructed recursively by the optimal solutions of these subnetworks. Based on the theorem we also present two algorithms to train a large ELM neural network efficiently: one is a parallel network training algorithm and the other is an incremental network training algorithm. The experimental results demonstrate the usefulness of the theorem and the validity of the developed algorithms.

Numerous state-of-the-art perceptual image quality assessment (IQA) algorithms share a common two-stage process: distortion description followed by distortion effects pooling. As for the first stage, the distortion descriptors or measurements are expected to be effective representatives of human visual variations, while the second stage should well express the relationship among quality descriptors and the perceptual visual quality. However, most of the existing quality descriptors (e.g., luminance, contrast, and gradient) do not seem to be consistent with human perception, and the effects pooling is often done in ad-hoc ways. In this paper, we propose a novel full-reference IQA metric. It applies non-negative matrix factorization (NMF) to measure image degradations by making use of the parts-based representation of NMF. On the other hand, a new machine learning technique [extreme learning machine (ELM)] is employed to address the limitations of the existing pooling techniques. Compared with neural networks and support vector regression, ELM can achieve higher learning accuracy with faster learning speed. Extensive experimental results demonstrate that the proposed metric has better performance and lower computational complexity in comparison with the relevant state-of-the-art approaches.

Dimension reduction is frequently adopted as a data preprocessing technique to facilitate data visualization, interpretation, and classification. Traditional dimension reduction methods such as linear discriminant analysis focus on maximizing the overall discrimination between all classes, which may be easily affected by outliers. To overcome this disadvantage, this paper proposes a novel method for multiclass dimension reduction, named dimension reduction by minimum error minimax probability machine (DR-MEMPM). It elaborately ensures that each pair of classes is well separated in the projected subspace by utilizing the separation probability between different pairwise classes. Therefore, it can put more emphasis on those less distinguishable classes, and the learned projection will not be dominated by some “outlier” classes which lie far away from other classes. We evaluate the proposed DR-MEMPM on a number of synthetic and real-world data sets, and show that it outperforms other state-of-the-art dimension reduction methods in terms of visual intuition and classification accuracy, especially when the distances between classes are unevenly distributed.

With the exponential growth of data and complexity of systems, fast machine learning/artificial intelligence and computational intelligence techniques are highly required. Many conventional computational intelligence techniques face bottlenecks in learning (e.g., intensive human intervention and convergence time) [item 1) in the Appendix]. However, efficient learning algorithms alternatively offer significant benefits including fast learning speed, ease of implementation, and minimal human intervention. The need for efficient and fast implementation of machine learning techniques in big data and dynamic varying systems poses many research challenges. This special issue highlights some latest development in the related areas.

People today rely more and more on global positioning system (GPS) for navigation when driving in unfamiliar environments. While GPS navigation is indispensable in an intelligent vehicle and provides convenience for road direction, concerns are also raised if the use of GPS may distract drivers to increase unnecessary stress. In this paper, we explore the effects of using GPS navigation on driver stress utilizing electrocardiogram (ECG) signals. In particular, the effects of higher or lower density of GPS instructions are studied. To analyze the driver stress, eight heart rate variability (HRV) features, which were commonly utilized in human stress related studies, were computed from ECG signals. Statistical significance tests were then performed to each HRV feature, so that those effective features for detecting driver stress may be localized. Our studies, based on road driving experiments with ten healthy subjects, showed that MeanRR, SDNN and HRVTri are the top three effective features to detect driver stress, while frequency domain features in general are not sensitive to driver stress. Based on the effective features, our analysis further showed that basically, driving with higher density of GPS instructions has no significant driver stress difference from driving with lower density of GPS instructions.

A smile is a common human facial expression used as the indicator for positive emotion. The detection of smiling has many applications, for example, controlling camera shutter when a smile is detected and measuring the degree of satisfaction during a video conference. Many feature extraction methods have been proposed for detecting a smile in the unconstrained scenarios. However, the dimensions of most existing feature descriptors are too huge, which limits their real applications. Moreover, features should be more effective to distinguish between smile face and non-smile face. Motivated by the observation that the mouth shape can effectively reflect a person's smile state, we extracted a novel and snappy set of features that form a feature vector named Pair-wise Distance Vector, which is calculated only based on few points around a mouth. After that, Extreme Learning Machine (ELM) is adopted to classify smile based on these features. The experimental results on GENKI-4K database show that our proposed method outperforms the state-of-the-art methods in terms of accuracy and dimension of features.

Monitoring the presence of occupants in a room in a timely manner is a fundamental step for effective building management. Environmental sensor networks have the advantages of high cost-efficiency and non-intrusiveness on privacy and are very suitable for room occupancy detection. Nonlinear discriminative models, e.g., support vector machine and neural networks, have shown good detection performance due to their ability to model complex relationship. However, they tend to produce unstable detection with frequent fluctuations over time, because they regard training data as independent and ignore the prior knowledge of the room occupancy, i.e., not changing very frequently. To improve the stability of the detection, we propose a two-stage structured learning approach with Extreme Learning Machine (ELM) as the local classifier. In the first stage, ELM is used as a fast nonlinear classifier to obtain preliminary detection results. In the second stage, we form data sequences consisting of the current and previous data points. The preliminary detection results by ELM of the data sequences are then used as input to a linear support vector machine for structured output to generate the final detection results. We test the proposed two-stage structured learning approach on a real-world dataset and show that the proposed approach outperforms the related machine learning methods.

Spatiotemporal fusion is important in providing high spatial resolution earth observations with a dense time series, and recently, learning-based fusion methods have been attracting broad interest. These algorithms project image patches onto a feature space with the enforcement of a simple mapping to predict the fine resolution patches from the corresponding coarse ones. However, the sophisticated projection, e.g., sparse representation, is always computationally complex and difficult to be implemented on large patches, which cannot grasp enough local structural information in the coarse patches. To address these issues, a novel spatiotemporal fusion method is proposed in this letter, using a powerful learning technique, i.e., extreme learning machine (ELM). Unlike traditional approaches, we devote to learning a mapping function on difference images directly, rather than the sophisticated feature representation followed by a simple mapping. Characterized by good generalization performance and fast speed, the ELM is employed to achieve accurate and fast fine patches prediction. The proposed algorithm is evaluated by five actual data sets of Landsat enhanced thematic mapper plus-moderate resolution imaging spectroradiometer acquisitions and experimental results show that our method obtains better fusion results while achieving much greater speed.

Polychronous neuronal group (PNG), a type of cell assembly, is one of the putative mechanisms for neural information representation. According to the reader-centric definition, some readout neurons can become selective to the information represented by polychronous neuronal groups under ongoing activity. Here, in computational models, we show that the frequently activated polychronous neuronal groups can be learned by readout neurons with joint weight-delay spike-timing-dependent plasticity. The identity of neurons in the group and their expected spike timing at millisecond scale can be recovered from the incoming weights and delays of the readout neurons. The detection performance can be further improved by two layers of readout neurons. In this way, the detection of polychronous neuronal groups becomes an intrinsic part of the network, and the readout neurons become differentiated members in the group to indicate whether subsets of the group have been activated according to their spike timing. The readout spikes representing this information can be used to analyze how PNGs interact with each other or propagate to downstream networks for higher-level processing.

The increasing demands of location-based services have spurred the rapid development of indoor positioning system and indoor localization system interchangeably (IPSs). However, the performance of IPSs suffers from noisy measurements. In this paper, two kinds of robust extreme learning machines (RELMs), corresponding to the close-to-mean constraint, and the small-residual constraint, have been proposed to address the issue of noisy measurements in IPSs. Based on whether the feature mapping in extreme learning machine is explicit, we respectively provide random-hidden-nodes and kernelized formulations of RELMs by second order cone programming. Furthermore, the computation of the covariance in feature space is discussed. Simulations and real-world indoor localization experiments are extensively carried out and the results demonstrate that the proposed algorithms can not only improve the accuracy and repeatability, but also reduce the deviation and worst case error of IPSs compared with other baseline algorithms.

Data may often contain noise or irrelevant information, which negatively affect the generalization capability of machine learning algorithms. The objective of dimension reduction algorithms, such as principal component analysis (PCA), non-negative matrix factorization (NMF), random projection (RP), and auto-encoder (AE), is to reduce the noise or irrelevant information of the data. The features of PCA (eigenvectors) and linear AE are not able to represent data as parts (e.g. nose in a face image). On the other hand, NMF and non-linear AE are maimed by slow learning speed and RP only represents a subspace of original data. This paper introduces a dimension reduction framework which to some extend represents data as parts, has fast learning speed, and learns the between-class scatter subspace. To this end, this paper investigates a linear and non-linear dimension reduction framework referred to as extreme learning machine AE (ELM-AE) and sparse ELM-AE (SELM-AE). In contrast to tied weight AE, the hidden neurons in ELM-AE and SELM-AE need not be tuned, and their parameters (e.g, input weights in additive neurons) are initialized using orthogonal and sparse random weights, respectively. Experimental results on USPS handwritten digit recognition data set, CIFAR-10 object recognition, and NORB object recognition data set show the efficacy of linear and non-linear ELM-AE and SELM-AE in terms of discriminative capability, sparsity, training time, and normalized mean square error.

Real-time driver distraction detection is the core to many distraction countermeasures and fundamental for constructing a driver-centered driver assistance system. While data-driven methods demonstrate promising detection performance, a particular challenge is how to reduce the considerable cost for collecting labeled data. This paper explored semi-supervised methods for driver distraction detection in real driving conditions to alleviate the cost of labeling training data. Laplacian support vector machine and semi-supervised extreme learning machine were evaluated using eye and head movements to classify two driver states: attentive and cognitively distracted. With the additional unlabeled data, the semi-supervised learning methods improved the detection performance (G-mean) by 0.0245, on average, over all subjects, as compared with the traditional supervised methods. As unlabeled training data can be collected from drivers' naturalistic driving records with little extra resource, semi-supervised methods, which utilize both labeled and unlabeled data, can enhance the efficiency of model development in terms of time and cost.

Extreme learning machine (ELM) is an emerging learning algorithm for the generalized single hidden layer feedforward neural networks, of which the hidden node parameters are randomly generated and the output weights are analytically computed. However, due to its shallow architecture, feature learning using ELM may not be effective for natural signals (e.g., images/videos), even with a large number of hidden nodes. To address this issue, in this paper, a new ELM-based hierarchical learning framework is proposed for multilayer perceptron. The proposed architecture is divided into two main components: 1) self-taught feature extraction followed by supervised feature classification and 2) they are bridged by random initialized hidden weights. The novelties of this paper are as follows: 1) unsupervised multilayer encoding is conducted for feature extraction, and an ELM-based sparse autoencoder is developed via l1 constraint. By doing so, it achieves more compact and meaningful feature representations than the original ELM; 2) by exploiting the advantages of ELM random feature mapping, the hierarchically encoded outputs are randomly projected before final decision making, which leads to a better generalization with faster learning speed; and 3) unlike the greedy layerwise training of deep learning (DL), the hidden layers of the proposed framework are trained in a forward manner. Once the previous layer is established, the weights of the current layer are fixed without fine-tuning. Therefore, it has much better learning efficiency than the DL. Extensive experiments on various widely used classification data sets show that the proposed algorithm achieves better and faster convergence than the existing state-of-the-art hierarchical learning methods. Furthermore, multiple applications in computer vision further confirm the generality and capability of the proposed learning scheme.

Due to the precise spike timing in neural coding, spiking neural network (SNN) possesses richer spatiotemporal dynamics compared to neural networks with firing rate coding. One of the distinct features of SNN, polychronous neuronal group (PNG), receives much attention from both computational neuroscience and machine learning communities. However, all existing algorithms detect PNGs from the spike recording collected after simulation in an offline manner. There is currently no algorithm that detects PNGs actually being activated in runtime (online manner), which could be potentially used as inputs to higher level neural processing. We propose a runtime detection algorithm particularly for activated PNGs, using PNG readout neurons, to fill this gap. The proposed algorithm can reveal the spatiotemporal PNG patterns embedded in spike trains, which is higher level neuronal dynamics. We demonstrate through an example that for composed input patterns, new PNGs except the constituent PNGs can be easily found using the proposed algorithm. As an important interpretation, we give further insights on how to use PNG readout neurons to construct layered network structure.

Learning representations from massive unlabeled data is a hot topic for high-level tasks in many applications. The recent great improvements on benchmark data sets, which are achieved by increasingly complex unsupervised learning methods and deep learning models with lots of parameters, usually require many tedious tricks and much expertise to tune. However, filters learned by these complex architectures are quite similar to standard hand-crafted features visually, and training the deep models costs quite long time to fine-tune their weights. In this paper, Extreme Learning Machine-Autoencoder (ELM-AE) is employed as the learning unit to learn local receptive fields at each layer, and the lower layer responses are transferred to the last layer (trans-layer) to form a more complete representation to retain more information. In addition, some beneficial methods in deep learning architectures such as local contrast normalization and whitening are added to the proposed hierarchical Extreme Learning Machine networks to further boost the performance. The obtained trans-layer representations are followed by block histograms with binary hashing to learn translation and rotation invariant representations, which are utilized to do high-level tasks such as recognition and detection. Compared to traditional deep learning methods, the proposed trans-layer representation method with ELM-AE based learning of local receptive filters has much faster learning speed and is validated in several typical experiments, such as digit recognition on MNIST and MNIST variations, object recognition on Caltech 101. State-of-the-art performances are achieved on both Caltech 101 15 samples per class task and 4 of 6 MNIST variations data sets, and highly impressive results are obtained on MNIST data set and other tasks.

In this paper, we proposed a driver drowsiness detection method for which only eyelid movement information was required. The proposed method consists of two major parts. 1) In order to obtain accurate eye openness estimation, a vision based eye openness recognition method was proposed to obtain an regression model that directly gave degree of eye openness from a low-resolution eye image without complex geometry modeling, which is efficient and robust to degraded image quality. 2) A novel feature extraction method based on unsupervised learning was also proposed to reveal hidden pattern from eyelid movements as well as reduce the feature dimension. The proposed method was evaluated and shown good performance.

Extreme learning machine (ELM) has recently attracted many researchers' interest due to its very fast learning speed, good generalization ability, and ease of implementation. It provides a unified solution that can be used directly to solve regression, binary, and multiclass classification problems. In this paper, we propose a stacked ELMs (S-ELMs) that is specially designed for solving large and complex data problems. The S-ELMs divides a single large ELM network into multiple stacked small ELMs which are serially connected. The S-ELMs can approximate a very large ELM network with small memory requirement. To further improve the testing accuracy on big data problems, the ELM autoencoder can be implemented during each iteration of the S-ELMs algorithm. The simulation results show that the S-ELMs even with random hidden nodes can achieve similar testing accuracy to support vector machine (SVM) while having low memory requirements. With the help of ELM autoencoder, the S-ELMs can achieve much better testing accuracy than SVM and slightly better accuracy than deep belief network (DBN) with much faster training speed.

Ship detection on spaceborne images has attracted great interest in the applications of maritime security and traffic control. Optical images stand out from other remote sensing images in object detection due to their higher resolution and more visualized contents. However, most of the popular techniques for ship detection from optical spaceborne images have two shortcomings: 1) Compared with infrared and synthetic aperture radar images, their results are affected by weather conditions, like clouds and ocean waves, and 2) the higher resolution results in larger data volume, which makes processing more difficult. Most of the previous works mainly focus on solving the first problem by improving segmentation or classification with complicated algorithms. These methods face difficulty in efficiently balancing performance and complexity. In this paper, we propose a ship detection approach to solving the aforementioned two issues using wavelet coefficients extracted from JPEG2000 compressed domain combined with deep neural network (DNN) and extreme learning machine (ELM). Compressed domain is adopted for fast ship candidate extraction, DNN is exploited for high-level feature representation and classification, and ELM is used for efficient feature pooling and decision making. Extensive experiments demonstrate that, in comparison with the existing relevant state-of-the-art approaches, the proposed method requires less detection time and achieves higher detection accuracy.

Extreme learning machine (ELM) is an efficient training algorithm originally proposed for single-hidden layer feedforward networks (SLFNs), of which the input weights are randomly chosen and need not to be fine-tuned. In this paper, we present a new stack architecture for ELM, to further improve the learning accuracy of ELM while maintaining its advantage of training speed. By exploiting the hidden information of ELM random feature space, a recovery-based training model is developed and incorporated into the proposed ELM stack architecture. Experimental results of the MNIST handwriting dataset demonstrate that the proposed algorithm achieves better and much faster convergence than the state-of-the-art ELM and deep learning methods.

Extreme learning machine (ELM) was initially proposed for single-hidden-layer feedforward neural networks (SLFNs). In the hidden layer (feature mapping), nodes are randomly generated independently of training data. Furthermore, a unified ELM was proposed, providing a single framework to simplify and unify different learning methods, such as SLFNs, least square support vector machines, proximal support vector machines, and so on. However, the solution of unified ELM is dense, and thus, usually plenty of storage space and testing time are required for large-scale applications. In this paper, a sparse ELM is proposed as an alternative solution for classification, reducing storage space and testing time. In addition, unified ELM obtains the solution by matrix inversion, whose computational complexity is between quadratic and cubic with respect to the training size. It still requires plenty of training time for large-scale problems, even though it is much faster than many other traditional methods. In this paper, an efficient training algorithm is specifically developed for sparse ELM. The quadratic programming problem involved in sparse ELM is divided into a series of smallest possible sub-problems, each of which are solved analytically. Compared with SVM, sparse ELM obtains better generalization performance with much faster training speed. Compared with unified ELM, sparse ELM achieves similar generalization performance for binary classification applications, and when dealing with large-scale binary classification problems, sparse ELM realizes even faster training speed than unified ELM.

In this paper, we propose a voting based online sequential extreme learning machine (VOS-ELM) for single hidden layer feedforward networks (SLFNs) to perform the online sequential multi-class classification. Utilizing the recent voting based extreme learning machine (V-ELM) and the online sequential extreme learning machine (OS-ELM), the newly developed VOS-ELM is able to classify online sequences by learning data one-by-one or chunk-by-chunk with fixed or varying chunk size and to reach a higher classification accuracy than the original OS-ELM. Simulations on several real world classification datasets show that VOS-ELM outperforms OS-ELM as well as several state-of-the-art online sequential algorithms.

Extreme learning machines (ELMs) have been proposed for generalized single-hidden-layer feedforward networks which need not be neuron alike and perform well in both regression and classification applications. The problem of determining the suitable network architectures is recognized to be crucial in the successful application of ELMs. This paper first proposes a dynamic ELM (D-ELM) where the hidden nodes can be recruited or deleted dynamically according to their significance to network performance, so that not only the parameters can be adjusted but also the architecture can be self-adapted simultaneously. Then, this paper proves in theory that such D-ELM using Lebesgue p-integrable hidden activation functions can approximate any Lebesgue p-integrable function on a compact input set. Simulation results obtained over various test problems demonstrate and verify that the proposed D-ELM does a good job reducing the network size while preserving good generalization performance.

Function approximators have been extensively used in Reinforcement Learning (RL) to deal with large or continuous space problems. However, batch learning Neural Networks (NN), one of the most common approximators, has been rarely applied to RL. In this paper, possible reasons for this are laid out and a solution is proposed. Specifically, a Receding Horizon Cache (RHC) structure is designed to collect training data for NN by dynamically archiving state-action pairs and actively updating their Q-values, which makes batch learning NN much easier to implement. Together with Extreme Learning Machine (ELM), a new RL with function approximation algorithm termed as RHC and ELM based RL (RHC-ELM-RL) is proposed. A mountain car task was carried out to test RHC-ELM-RL and compare its performance with other algorithms.

We consider the problem of intrusion detection in a computer network, and investigate the use of extreme learning machines (ELMs) to classify and detect the intrusions. With increasing connectivity between networks, the risk of information systems to external attacks or intrusions has increased tremendously. Machine learning methods like support vector machines (SVMs) and neural networks have been widely used for intrusion detection. These methods generally suffer from long training times, require parameter tuning, or do not perform well in multi-class classification. We propose a basic ELM method based on random features, and a kernel based ELM method for classification. We compare our methods with commonly used SVM techniques in both binary and multi-class classifications. Simulation results show that the proposed basic ELM approach outperforms SVM in training and testing speed, while the proposed kernel based ELM achieves higher detection accuracy than SVM in multi-class classification case.

Credit risk evaluation has become an increasingly important field in financial risk management for financial institutions, especially for banks and credit card companies. Many data mining and statistical methods have been applied to this field. Extreme learning machine (ELM) classifier as a type of generalized single hidden layer feed-forward networks has been used in many applications and achieve good classification accuracy. Thus, we use ELM (kernel based) as a classification tool to perform the credit risk evaluation in this paper. The simulations are done on two credit risk evaluation datasets with three different kernel functions. Simulation results show that the kernel based ELM is more suitable for credit risk evaluation than the popular used Support Vector Machines (SVMs) with consideration of overall, good and bad accuracies.

Traditional risk score prediction is based on vital signs and clinical assessment. In this paper, we present an intelligent scoring system for the prediction of cardiac arrest within 72 h. The patient population is represented by a set of feature vectors, from which risk scores are derived based on geometric distance calculation and support vector machine. Each feature vector is a combination of heart rate variability (HRV) parameters and vital signs. Performance evaluation is conducted on the leave-one-out cross-validation framework, and receiver operating characteristic, sensitivity, specificity, positive predictive value, and negative predictive value are reported. Experimental results reveal that the proposed scoring system not only achieves satisfactory performance on determining the risk of cardiac arrest within 72 h but also has the ability to generate continuous risk scores rather than a simple binary decision by a traditional classifier. Furthermore, the proposed scoring system works well for both balanced and imbalanced datasets, and the combination of HRV parameters and vital signs shows superiority in prediction to using HRV parameters only or vital signs only.

Extreme learning machines (ELMs) have been proposed for generalized single-hidden-layer feedforward networks which need not be neuron-like and perform well in both regression and classification applications. In this brief, we propose an ELM with adaptive growth of hidden nodes (AG-ELM), which provides a new approach for the automated design of networks. Different from other incremental ELMs (I-ELMs) whose existing hidden nodes are frozen when the new hidden nodes are added one by one, in AG-ELM the number of hidden nodes is determined in an adaptive way in the sense that the existing networks may be replaced by newly generated networks which have fewer hidden nodes and better generalization performance. We then prove that such an AG-ELM using Lebesgue p-integrable hidden activation functions can approximate any Lebesgue p-integrable function on a compact input set. Simulation results demonstrate and verify that this new approach can achieve a more compact network architecture than the I-ELM.

The online backpropagation (BP) training procedure has been extensively explored in scientific research and engineering applications. One of the main factors affecting the performance of the online BP training is the learning rate. This paper proposes a new dynamic learning rate which is based on the estimate of the minimum error. The global convergence theory of the online BP training procedure with the proposed learning rate is further studied. It is proved that: 1) the error sequence converges to the global minimum error; and 2) the weight sequence converges to a fixed point at which the error function attains its global minimum. The obtained global convergence theory underlies the successful applications of the online BP training procedure. Illustrative examples are provided to support the theoretical analysis.

Due to the simplicity of their implementations, least square support vector machine (LS-SVM) and proximal support vector machine (PSVM) have been widely used in binary classification applications. The conventional LS-SVM and PSVM cannot be used in regression and multiclass classification applications directly, although variants of LS-SVM and PSVM have been proposed to handle such cases. This paper shows that both LS-SVM and PSVM can be simplified further and a unified learning framework of LS-SVM, PSVM, and other regularization algorithms referred to extreme learning machine (ELM) can be built. ELM works for the "generalized" single-hidden-layer feedforward networks (SLFNs), but the hidden layer (or called feature mapping) in ELM need not be tuned. Such SLFNs include but are not limited to SVM, polynomial network, and the conventional feedforward neural networks. This paper shows the following: 1) ELM provides a unified learning platform with a widespread type of feature mappings and can be applied in regression and multiclass classification applications directly; 2) from the optimization method point of view, ELM has milder optimization constraints compared to LS-SVM and PSVM; 3) in theory, compared to ELM, LS-SVM and PSVM achieve suboptimal solutions and require higher computational complexity; and 4) in theory, ELM can approximate any target continuous function and classify any disjoint regions. As verified by the simulation results, ELM tends to have better scalability and achieve similar (for regression and binary class cases) or much better (for multiclass cases) generalization performance at much faster learning speed (up to thousands times) than traditional SVM and LS-SVM.

In this paper, a weighting-delay-based method is developed for the study of the stability problem of a class of recurrent neural networks (RNNs) with time-varying delay. Different from previous results, the delay interval [0, d(t)] is divided into some variable subintervals by employing weighting delays. Thus, new delay-dependent stability criteria for RNNs with time-varying delay are derived by applying this weighting-delay method, which are less conservative than previous results. The proposed stability criteria depend on the positions of weighting delays in the interval [0, d(t)] , which can be denoted by the weighting-delay parameters. Different weighting-delay parameters lead to different stability margins for a given system. Thus, a solution based on optimization methods is further given to calculate the optimal weighting-delay parameters. Several examples are provided to verify the effectiveness of the proposed criteria.

This paper studies a class of new neural networks referred to as switched neutral-type neural networks (SNTNNs) with time-varying delays, which combines switched systems with a class of neutral-type neural networks. The less conservative robust stability criteria for SNTNNs with time-varying delays are proposed by using a new Lyapunov-Krasovskii functional and a novel series compensation (SC) technique. Based on the new functional, SNTNNs with fast-varying neutral-type delay (the derivative of delay is more than one) is first considered. The benefit brought by employing the SC technique is that some useful negative definite elements can be included in stability criteria, which are generally ignored in the estimation of the upper bound of derivative of Lyapunov-Krasovskii functional in literature. Furthermore, the criteria proposed in this paper are also effective and less conservative in switched recurrent neural networks which can be considered as special cases of SNTNNs. The simulation results based on several numerical examples demonstrate the effectiveness of the proposed criteria.

This paper is concerned with the robust exponential synchronization problem of a class of chaotic delayed neural networks with different parametric uncertainties. A novel impulsive control scheme (so-called dual-stage impulsive control) is proposed. Based on the theory of impulsive functional differential equations, a global exponential synchronization error bound together with some new sufficient conditions expressed in the form of linear matrix inequalities (LMIs) is derived in order to guarantee that the synchronization error dynamics can converge to a predetermined level. Furthermore, to estimate the stable region, a novel optimization control algorithm is established, which can deal with the minimum problem with two nonlinear terms coexisting in LMIs effectively. The idea and approach developed in this paper can provide a more practical framework for the synchronization of multiperturbation delayed chaotic systems. Simulation results finally demonstrate the effectiveness of the proposed method.

Online Sequential Extreme Learning Machine (OS-ELM) proposed by Liang et al [1] is a faster and more accurate online sequential learning algorithm as compared to other current sequential algorithms. It can learn data one-by-one or chunk-by-chunk with fixed or varying chunk size. However, there is one of the remaining challenges for OS-ELM that it could not determine the optimal network structure automatically. In this paper, we propose a Constructive Enhancement for OS-ELM (CEOS-ELM), which can add random hidden nodes one-by-one or group-by-group with fixed or varying group size. CEOS-ELM is searching for the optimal network architecture during the sequential learning process, and it can handle both additive and radial basis function (RBF) hidden nodes. The optimal number of hidden nodes can be obtained automatically after training. The simulation results show that with CEOS-ELM, the network can achieve comparable generalization performance with OS-ELM and more compact network structure.

One of the open problems in neural network research is how to automatically determine network architectures for given applications. In this brief, we propose a simple and efficient approach to automatically determine the number of hidden nodes in generalized single-hidden-layer feedforward networks (SLFNs) which need not be neural alike. This approach referred to as error minimized extreme learning machine (EM-ELM) can add random hidden nodes to SLFNs one by one or group by group (with varying group size). During the growth of the networks, the output weights are updated incrementally. The convergence of this approach is proved in this brief as well. Simulation results demonstrate and verify that our new approach is much faster than other sequential/incremental/growing algorithms with good generalization performance.

In this correspondence, an online sequential fuzzy extreme learning machine (OS-Fuzzy-ELM) has been developed for function approximation and classification problems. The equivalence of a Takagi-Sugeno-Kang (TSK) fuzzy inference system (FIS) to a generalized single hidden-layer feedforward network is shown first, which is then used to develop the OS-Fuzzy-ELM algorithm. This results in a FIS that can handle any bounded nonconstant piecewise continuous membership function. Furthermore, the learning in OS-Fuzzy-ELM can be done with the input data coming in a one-by-one mode or a chunk-by-chunk (a block of data) mode with fixed or varying chunk size. In OS-Fuzzy-ELM, all the antecedent parameters of membership functions are randomly assigned first, and then, the corresponding consequent parameters are determined analytically. Performance comparisons of OS-Fuzzy-ELM with other existing algorithms are presented using real-world benchmark problems in the areas of nonlinear system identification, regression, and classification. The results show that the proposed OS-Fuzzy-ELM produces similar or better accuracies with at least an order-of-magnitude reduction in the training time.

In this paper, Extreme Learning Machine (ELM) is introduced to predict the subcellular localization of proteins based on the frequent subsequences. It is proved that ELM is extremely fast and can provide good generalization performance. We evaluated the performance of ELM on four localization sites with frequent subsequences as the feature space. A new parameter called Comparesup was introduced to help the feature selection. The performance of ELM was tested on data with different number of frequent subsequences, which were determined by different range of Comparesup. The results demonstrated that ELM performed better than previously reported results, for all of the four localization sites.

In the paper, the multi-class pattern classification using extreme learning machine (ELM) is studied. The study is based on either a series of ELM binary classifiers or a single ELM classifier. When using binary ELM classifiers, the multi-class problem is decomposed into two-class problem using the one-against-all (OAA) and one-against-one (OAO) schemes, which are named as ELM-OAA and ELM-OAO respectively for brevity. In a single ELM classifier, the multi-class problem is implemented with an architecture of multi-output nodes which is equal to the number of pattern classes. Their performance is evaluated using some multi-class benchmark problems and simulation results show that ELM-OAA and ELM-OAO requires fewer hidden nodes than the single ELM classifier. In addition ELM-OAO usually has similar or less computation burden than the single ELM classifier when the pattern class labels is not larger than 10.

In this reply, we refer to Wang and Wan's comments on our research publication. We found that the comment letter contains some inaccurate statements. The comment letter contains some contradictions as well.

In this paper, the recently developed Extreme Learning Machine (ELM) is used for direct multicategory classification problems in the cancer diagnosis area. ELM avoids problems like local minima, improper learning rate and overfitting commonly faced by iterative learning methods and completes the training very fast. We have evaluated the multi-category classification performance of ELM on three benchmark microarray datasets for cancer diagnosis, namely, the GCM dataset, the Lung dataset and the Lymphoma dataset. The results indicate that ELM produces comparable or better classification accuracies with reduced training time and implementation complexity compared to artificial neural networks methods like conventional back-propagation ANN, Linder's SANN, and Support Vector Machine methods like SVM-OVO and Ramaswamy's SVM-OVA. ELM also achieves better accuracies for classification of individual categories.

This paper presents a fuzzy control strategy for aircraft autolanding under the failures of stuck control surfaces and severe winds. The control strategy incorporates a TSK fuzzy neural network implementing TSK fuzzy model and it aids an existing conventional controller called a Baseline Trajectory Following Controller (BTFC). The TSK fuzzy neural network is trained by the Online Sequential Fuzzy Extreme Learning Machine (Fuzzy-ELM) algorithm. In Fuzzy-ELM algorithm the parameters of fuzzy membership functions need not be adjusted during training and one may simply randomly assign values to them. Performance of the proposed fuzzy control scheme is evaluated for a typical aircraft autolanding with a double failure of left elevator and left aileron stuck at different deflections. The results indicate superior performance of the proposed fuzzy fault tolerant controller.

The problem of terrain modeling is basically a type of function approximation problem. This type of problem has been widely studied in the soft computing community. In recent years, neural networks have been successfully applied to surface reconstruction and classification problems involving scattered data. However, due to the iterative nature of training a neural network, the resulting high cost in computational time limits the implementation of machine learning based methods in many real world applications (for example, navigation applications in unmanned aerial vehicles) that require fast generation of terrain models. A recently proposed machine learning method, the extreme learning machine (ELM), is able to train single-layer feed forward neural networks with excellent speed and good generalization. In this paper, we present terrain modeling using various machine learning methods, and we compare the performances of these methods with ELM. We also present a comparison of terrain modeling performances between ELM and the popular choice of terrain and surface modeling technique, the Delaunay triangulation with linear interpolation. Our results show that machine learning using ELM offers a potential solution to terrain modeling problems with good performances

In this paper, a recently developed machine learning algorithm referred to as Extreme Learning Machine (ELM) is used to classify five mental tasks from different subjects using electroencephalogram (EEG) signals available from a well-known database. Performance of ELM is compared in terms of training time and classification accuracy with a Backpropagation Neural Network (BPNN) classifier and also Support Vector Machines (SVMs). For SVMs, the comparisons have been made for both 1-against-1 and 1-against-all methods. Results show that ELM needs an order of magnitude less training time compared with SVMs and two orders of magnitude less compared with BPNN. The classification accuracy of ELM is similar to that of SVMs and BPNN. The study showed that smoothing of the classifiers' outputs can significantly improve their classification accuracies.

Terrain models that permit multiresolution access are essential for model predictive control of unmanned aerial vehicles in low-level flights. The authors present the extreme learning machine (ELM), a recently proposed learning paradigm, as a mechanism for learning the stored digital elevation information to allow multiresolution access. We give results of simulations designed to compare the performance of our approach with two other approaches for multiresolution access, namely: 1) linear interpolation on Delaunay triangles of the sampled terrain data points and 2) terrain learning using support vector machines (SVMs). The results show that to achieve the same mean square error during access, the memory needed in our approach is significantly lower. Additionally, the offline training time for the ELM network is much less than that for the SVM.

According to conventional neural network theories, single-hidden-layer feedforward networks (SLFNs) with additive or radial basis function (RBF) hidden nodes are universal approximators when all the parameters of the networks are allowed adjustable. However, as observed in most neural network implementations, tuning all the parameters of the networks may cause learning complicated and inefficient, and it may be difficult to train networks with nondifferential activation functions such as threshold networks. Unlike conventional neural network theories, this paper proves in an incremental constructive method that in order to let SLFNs work as universal approximators, one may simply randomly choose hidden nodes and then only need to adjust the output weights linking the hidden layer and the output layer. In such SLFNs implementations, the activation functions for additive nodes can be any bounded nonconstant piecewise continuous functions g : R --> R and the activation functions for RBF nodes can be any integrable piecewise continuous functions g : R --> R and integral of R g(x)dx not equal to 0. The proposed incremental method is efficient not only for SFLNs with continuous (including nondifferentiable) activation functions but also for SLFNs with piecewise continuous (such as threshold) activation functions. Compared to other popular methods such a new network is fully automatic and users need not intervene the learning process by manually tuning control parameters.

In some practical applications of neural networks, fast response to external events within an extremely short time is highly demanded and expected. However, the extensively used gradient-descent-based learning algorithms obviously cannot satisfy the real-time learning needs in many applications, especially for large-scale applications and/or when higher generalization performance is required. Based on Huang's constructive network model, this paper proposes a simple learning algorithm capable of real-time learning which can automatically select appropriate values of neural quantizers and analytically determine the parameters (weights and bias) of the network at one time only. The performance of the proposed algorithm has been systematically investigated on a large batch of benchmark real-world regression and classification problems. The experimental results demonstrate that our algorithm can not only produce good generalization performance but also have real-time learning and prediction capability. Thus, it may provide an alternative approach for the practical applications of neural networks where real-time learning and prediction implementation is required.

In this paper, we develop an online sequential learning algorithm for single hidden layer feedforward networks (SLFNs) with additive or radial basis function (RBF) hidden nodes in a unified framework. The algorithm is referred to as online sequential extreme learning machine (OS-ELM) and can learn data one-by-one or chunk-by-chunk (a block of data) with fixed or varying chunk size. The activation functions for additive nodes in OS-ELM can be any bounded nonconstant piecewise continuous functions and the activation functions for RBF nodes can be any integrable piecewise continuous functions. In OS-ELM, the parameters of hidden nodes (the input weights and biases of additive nodes or the centers and impact factors of RBF nodes) are randomly selected and the output weights are analytically determined based on the sequentially arriving data. The algorithm uses the ideas of ELM of Huang et al. developed for batch learning which has been shown to be extremely fast with generalization performance better than other batch training methods. Apart from selecting the number of hidden nodes, no other control parameters have to be manually chosen. Detailed performance comparison of OS-ELM is done with other popular sequential learning algorithms on benchmark problems drawn from the regression, classification and time series prediction areas. The results show that the OS-ELM is faster than the other sequential algorithms and produces better generalization performance.

Traditionally, two protein sequences are classified into the same class if they have high homology in terms of feature patterns extracted through sequence alignment algorithms. These algorithms compare an unseen protein sequence with all the identified protein sequences and returned the higher scored protein sequences. As the sizes of the protein sequence databases are very large, it is a very time consuming job to perform exhaustive comparison of existing protein sequence. Therefore, there is a need to build an improved classification system for effectively identifying protein sequences. In this paper, a recently developed machine learning algorithm referred to as the extreme learning machine (ELM) is used to classify protein sequences with ten classes of super-families downloaded from a public domain database. A comparative study on system performance is conducted between ELM and the main conventional neural network classifier - backpropagation neural networks. Results show that ELM needs up to four orders of magnitude less training time compared to BP Network. The classification accuracy of ELM is also higher than that of BP network. For given network architecture, ELM does not have any control parameters (i.e, stopping criteria, learning rate, learning epoches, etc.) to be manually tuned and can be implemented easily.

This paper investigates the performance of the latest developed GGAP-RBF network in time series prediction applications. The growing and pruning strategy of GGAP-RBF are based on linking the required learning accuracy with the significance of the nearest added new neuron. Significance of a neuron is a measure of the average information content of that neuron. GGAP-RBF algorithm may be attractive in real time-series applications due to its good efficiency and simple topology. This paper investigates its performance in two important real time-series applications: predictions of Nasdaq stock and weekly nitrate contamination of drinking water. The simulation results demonstrate that GGAP-RBF network can achieve good prediction accuracy in an efficient and easy way.

In this letter, a new nonlinear approach based on a combination of the fuzzy c-means clustering (FCMC), feature vector selection and principal component analysis (PCA) is proposed to extract features of multispectral images when a very large number of samples need to be processed. The main contribution of this letter is to provide a preprocessing method for classifying these images with higher accuracy compared to the single PCA and kernel PCA. Finally, some experimental results demonstrate that our proposed approach is effective and efficient in analyzing multispectral images.

Support vector machines (SVMs) have been extensively used. However, it is known that SVMs face difficulty in solving large complex problems due to the intensive computation involved in their training algorithms, which are at least quadratic with respect to the number of training examples. This paper proposes a new, simple, and efficient network architecture which consists of several SVMs each trained on a small subregion of the whole data sampling space and the same number of simple neural quantizer modules which inhibit the outputs of all the remote SVMs and only allow a single local SVM to fire (produce actual output) at any time. In principle, this region-computing based modular network method can significantly reduce the learning time of SVM algorithms without sacrificing much generalization performance. The experiments on a few real large complex benchmark problems demonstrate that our method can be significantly faster than single SVMs without losing much generalization performance.

This paper presents a new sequential learning algorithm for radial basis function (RBF) networks referred to as generalized growing and pruning algorithm for RBF (GGAP-RBF). The paper first introduces the concept of significance for the hidden neurons and then uses it in the learning algorithm to realize parsimonious networks. The growing and pruning strategy of GGAP-RBF is based on linking the required learning accuracy with the significance of the nearest or intentionally added new neuron. Significance of a neuron is a measure of the average information content of that neuron. The GGAP-RBF algorithm can be used for any arbitrary sampling density for training samples and is derived from a rigorous statistical point of view. Simulation results for bench mark problems in the function approximation area show that the GGAP-RBF outperforms several other sequential learning algorithms in terms of learning speed, network size and generalization performance regardless of the sampling density function of the training data.

The central problem in training a radial basis function neural network is the selection of hidden layer neurons. In this paper, we propose to select hidden layer neurons based on data structure preserving criterion. Data structure denotes relative location of samples in the high-dimensional space. By preserving the data structure of samples including those that are close to separation boundaries between different classes, the neuron subset selected retains the separation margin underlying the full set of hidden layer neurons. As a direct result, the network obtained tends to generalize well.

A new learning algorithm called extreme learning machine (ELM) has recently been proposed for single-hidden layer feedforward neural networks (SLFNs) to easily achieve good generalization performance at extremely fast learning speed. ELM randomly chooses the input weights and analytically determines the output weights of SLFNs. This paper shows that ELM can be extended to radial basis function (RBF) network case, which allows the centers and impact widths of RBF kernels to be randomly generated and the output weights to be simply analytically calculated instead of iteratively tuned. Interestingly, the experimental results show that the ELM algorithm for RBF networks can complete learning at extremely fast speed and produce generalization performance very close to that of SVM in many artificial and real benchmarking function approximation and classification problems. Since ELM does not require validation and human-intervened parameters for given network architectures, ELM can be easily used.

The gradient-based learning algorithms are usually used to train feedforward neural networks. In these algorithms, the parameters of the network are adjusted iteratively according to the partial gradients of the user-defined performance functions. Such algorithms usually require tens to hundreds of learning epochs to reach the required accuracy. If it sticks in the local minimum in the learning process, the situation tends to be even worse. In Huang et al., a novel fast learning algorithm called extreme learning machine (ELM) for single-hidden-layer neural networks (SLFNs) has been proposed where a constructive method is used instead of a gradient-based learning algorithm. In this paper, we further verify the performance of ELM on two benchmark artificial problems.

Nowdays, neural networks have been widely used. In some areas, such as bioinfomatics and data mining, the problems are usually large-scaled and time-concerned. Thus, the requirement of fast learning algorithm and network architecture for large scale problems is stressed. In this paper, a novel network architecture consisting of several network modules is proposed. In this architecture, each network module is trained to learn a subset of the whole training data, and a number of neural quantizers are used to activate the corresponding network module to output. In principle, this modular network architecture (MNA) can effectively reduce the learning time and achieve the similar gerneralization performance. The results of experiments on several real-world benchmarking problems shown here to make the algorithm convincible.

It is clear that the learning speed of feedforward neural networks is in general far slower than required and it has been a major bottleneck in their applications for past decades. Two key reasons behind may be: 1) the slow gradient-based learning algorithms are extensively used to train neural networks, and 2) all the parameters of the networks are tuned iteratively by using such learning algorithms. Unlike these traditional implementations, this paper proposes a new learning algorithm called extreme learning machine (ELM) for single-hidden layer feedforward neural networks (SLFNs) which randomly chooses the input weights and analytically determines the output weights of SLFNs. In theory, this algorithm tends to provide the best generalization performance at extremely fast learning speed. The experimental results based on real-world benchmarking function approximation and classification problems including large complex applications show that the new algorithm can produce best generalization performance in some cases and can learn much faster than traditional popular learning algorithms for feedforward neural networks.

We summarize some of the key research areas in brain sciences and neural networks that have recently been or are being worked on by researchers in Singapore. Researchers in Singapore are developing theory of neural networks, notably improved radial basis function networks, fuzzy neural networks, and fast learning neural networks. Applications of neural networks include bioinformatics, multimedia, data mining, and communications. Researchers are also working with neurophysiologists on functional brain imaging and brain disease analysis.

GAP-RBF (growing and pruning RBF) algorithm is a newly developed sequential growing and pruning algorithm for RBF networks for function approximation problems. It has been confirmed to produce excellent performance for problems in function approximation area, but its performance for classification problems has not been evaluated yet. In this paper, the performance of GAP-RBF for bio-medical classification problems is investigated. Its classification performance is compared with the conventional multilayer feed forward network (MFN) and a well-known sequential learning algorithm-minimal resource allocation network (MRAN) based on two benchmark problems from the bio-medical classification area from PROBEN1 database. The results indicate that GAP-RBF/ algorithm can achieve a higher or at least similar classification accuracy with a more compact network structure and faster learning speed. Some limitations of this algorithm for classification problems are also identified.

This paper presents a simple sequential growing and pruning algorithm for radial basis function (RBF) networks. The algorithm referred to as growing and pruning (GAP)-RBF uses the concept of "Significance" of a neuron and links it to the learning accuracy. "Significance" of a neuron is defined as its contribution to the network output averaged over all the input data received so far. Using a piecewise-linear approximation for the Gaussian function, a simple and efficient way of computing this significance has been derived for uniformly distributed input data. In the GAP-RBF algorithm, the growing and pruning are based on the significance of the "nearest" neuron. In this paper, the performance of the GAP-RBF learning algorithm is compared with other well-known sequential learning algorithms like RAN, RANEKF, and MRAN on an artificial problem with uniform input distribution and three real-world nonuniform, higher dimensional benchmark problems. The results indicate that the GAP-RBF algorithm can provide comparable generalization performance with a considerably reduced network size and training time.

The problem of the necessary complexity of neural networks is of interest in applications. In this paper, learning capability and storage capacity of feedforward neural networks are considered. We markedly improve the recent results by introducing neural-network modularity logically. This paper rigorously proves in a constructive method that two-hidden-layer feedforward networks (TLFNs) with 2/spl radic/(m+2)N (/spl Lt/N) hidden neurons can learn any N distinct samples (x/sub i/, t/sub i/) with any arbitrarily small error, where m is the required number of output neurons. It implies that the required number of hidden neurons needed in feedforward networks can be decreased significantly, comparing with previous results. Conversely, a TLFN with Q hidden neurons can store at least Q/sup 2//4(m+2) any distinct data (x/sub i/, t/sub i/) with any desired precision.

Multilayer perceptrons with hard-limiting (signum) activation functions can form complex decision regions. It is well known that a three-layer perceptron (two hidden layers) can form arbitrary disjoint decision regions and a two-layer perceptron (one hidden layer) can form single convex decision regions. This paper further proves that single hidden layer feedforward neural networks (SLFN's) with any continuous bounded nonconstant activation function or any arbitrary bounded (continuous or not continuous) activation function which has unequal limits at infinities (not just perceptrons) can form disjoint decision regions with arbitrary shapes in multidimensional cases. SLFN's with some unbounded activation function can also form disjoint decision regions with arbitrary shapes.

It has been proved that in one-dimensional cases, the weights of Kohonen's self-organizing maps (SOM) will become ordered with probability 1; once the weights are ordered, they cannot become disordered in future training. It is difficult to analyze Kohonen's SOMs in multidimensional cases; however, it has been conjectured that similar results seem to be obtainable in multidimensional cases. In this note, we show that in multidimensional cases, even though the weights are ordered at some time, it is possible that they become disordered in the future.

It is well known that standard single-hidden layer feedforward networks (SLFNs) with at most N hidden neurons (including biases) can learn N distinct samples (x(i),t(i)) with zero error, and the weights connecting the input neurons and the hidden neurons can be chosen "almost" arbitrarily. However, these results have been obtained for the case when the activation function for the hidden neurons is the signum function. This paper rigorously proves that standard single-hidden layer feedforward networks (SLFNs) with at most N hidden neurons and with any bounded nonlinear activation function which has a limit at one infinity can learn N distinct samples (x(i),t(i)) with zero error. The previous method of arbitrarily choosing weights is not feasible for any SLFN. The proof of our result is constructive and thus gives a method to directly find the weights of the standard SLFNs with any such bounded nonlinear activation function as opposed to iterative training algorithms in the literature.

In the above paper Chen et al. investigated the capability of uniformly approximating functions in C(Rn) by standard feedforward neural networks. They found that the boundedness condition on the sigmoidal function plays an essential role in the approximation, and conjectured that the boundedness of the sigmoidal function is a necessary and sufficient condition for the validity of the approximation theorem. However, we find that the conjecture is not correct, that is, the boundedness condition is not sufficient or necessary in C(Rn). Instead, boundedness and unequal limits at infinities conditions on the activation functions are sufficient, but not necessary in C(Rn).