Data augmentation is widely utilized as an effective technique to enhance the generalization performance of deep models. However, data augmentation may inevitably introduce distribution shifts and noises, which significantly constrain the potential and deteriorate the performance of deep networks. To this end, we propose a novel information-preserving framework, namely IPF-RDA, to enhance the robustness of data augmentations in this paper. IPF-RDA combines the proposal of (i) a new class-discriminative information estimation algorithm that identifies the points most vulnerable to data augmentation operations and corresponding importance scores; And (ii) a new information-preserving scheme that preserves the critical information in the augmented samples and ensures the diversity of augmented data adaptively. We divide data augmentation methods into three categories according to the operation types and integrate these approaches into our framework accordingly. After being integrated into our framework, the robustness of data augmentation methods can be enhanced and their full potential can be unleashed. Extensive experiments demonstrate that although being simple, IPF-RDA consistently improves the performance of numerous commonly used state-of-the-art data augmentation methods with popular deep models on a variety of datasets, including CIFAR-10, CIFAR-100, Tiny-ImageNet, CUHK03, Market1501, Oxford Flower, and MNIST, where its performance and scalability are stressed.

Large-scale datasets have been pivotal to the advancements of deep learning models in recent years, but training on such large datasets inevitably incurs substantial storage and computational overhead. Meanwhile, real-world datasets often contain redundant and noisy data, imposing a negative impact on training efficiency and model performance. Data selection has shown promise in identifying the most representative samples from the entire dataset, which aims to minimize the performance gap with reduced training costs. Existing works typically rely on single-modality information to assign importance scores for individual samples, which may lead to inaccurate assessments, especially when dealing with noisy or corrupted samples. To address this limitation, we propose a novel CLIP-powered data selection framework that leverages multimodal information for more robust and generalizable sample selection. Specifically, our framework consists of three key modules—dataset adaptation, sample scoring, and selection optimization—that together harness extensive pre-trained multimodal knowledge to comprehensively assess sample influence and optimize the selection results through multi-objective optimization. Extensive experiments demonstrate that our approach consistently outperforms existing state-of-the-art baselines on various benchmark datasets. Notably, our method effectively removes noisy or damaged samples from the dataset, enabling it to achieve even higher performance with less data. This indicates that it is not only a way to accelerate training but can also improve overall data quality. The implementation is available at https://github.com/Jackbrocp/clip-powered-data-selection.

Dynamic data selection aims to accelerate training with lossless performances. However, reducing training data inherently limits data diversity, potentially hindering generalization. While data augmentation is widely used to enhance diversity, it is typically not optimized in conjunction with selection. As a result, directly combining these techniques fails to fully exploit their synergies. To tackle the challenge, we propose a novel online data training framework that, for the first time, unifies dynamic data selection and augmentation, achieving both training efficiency and enhanced performance. Our method estimates each sample’s joint distribution of local density and multimodal semantic consistency, allowing for the targeted selection of augmentation-suitable samples while suppressing the inclusion of noisy or ambiguous data. This enables a more significant reduction in dataset size without sacrificing model generalization. Experimental results demonstrate that our method outperforms existing state-of-the-art approaches on various benchmark datasets and architectures, e.g., reducing 50% training costs on ImageNet-1k with lossless performance. Furthermore, our approach enhances noise resistance and improves model robustness, reinforcing its practical utility in real-world scenarios.

Continual learning requires machine learning models to continuously acquire new knowledge in dynamic environments while avoiding the forgetting of previous knowledge. Prompt-based continual learning methods effectively address the issue of catastrophic forgetting through prompt expansion and selection. However, existing approaches often suffer from low accuracy in prompt selection, which can result in the model receiving biased knowledge and making biased predictions. To address this issue, we propose the Multiple Queries with Multiple Keys (MQMK) prompt matching paradigm for precise prompt selection. The goal of MQMK is to select the prompts whose training data distribution most closely matches that of the test sample. Specifically, Multiple Queries enable precise breadth search by introducing task-specific knowledge, while Multiple Keys perform deep search by representing the feature distribution of training samples at a fine-grained level. Each query is designed to perform local matching with a designated task to reduce interference across queries. Experiments show that MQMK enhances the prompt matching rate by over 30\% in challenging scenarios and achieves state-of-the-art performance on three widely adopted continual learning benchmarks. The code is available at https://github.com/DunweiTu/MQMK.

In this paper, a control method of double-swing-arm tracked robot based on deep reinforcement learning is proposed to solve the problem of stable operation of the robot on uneven terrain. A control algorithm without complex kinematics analysis is designed, so that the robot can learn independently and keep balance on various irregular terrain. This paper mainly studies the stability of the robot when crossing the terrain, so as to reduce the damage to the robot hardware. The main contributions are as follows: (1) Combining the hierarchical control strategy with the curiosity module, and testing in the simulation environment has achieved good performance; (2) By adding stability design, the robot can pass through uneven terrain more smoothly; (3) Three kinds of terrain task scenes are developed in the simulation environment, and the effectiveness of the control algorithm is verified. The experimental results show that this method can effectively improve the robot’s ability to cross complex terrain while maintaining high stability.

Data augmentation (DA) is widely employed to improve the generalization performance of deep models. However, most existing DA methods employ augmentation operations with fixed or random magnitudes throughout the training process. While this fosters data diversity, it can also inevitably introduce uncontrolled variability in augmented data, which could potentially cause misalignment with the evolving training status of the target models. Both theoretical and empirical findings suggest that this misalignment increases the risks of both underfitting and overfitting. To address these limitations, we propose AdaAugment, an innovative and tuning-free adaptive augmentation method that leverages reinforcement learning to dynamically and adaptively adjust augmentation magnitudes for individual training samples based on real-time feedback from the target network. Specifically, AdaAugment features a dual-model architecture consisting of a policy network and a target network, which are jointly optimized to adapt augmentation magnitudes in accordance with the model's training progress effectively. The policy network optimizes the variability within the augmented data, while the target network utilizes the adaptively augmented samples for training. These two networks are jointly optimized and mutually reinforce each other. Extensive experiments across benchmark datasets and deep architectures demonstrate that AdaAugment consistently outperforms other state-of-the-art DA methods in effectiveness while maintaining remarkable efficiency. Code is available at https://github.com/Jackbrocp/AdaAugment.

Data augmentation has been widely used in low-resource NER tasks to tackle the problem of data sparsity. However, previous data augmentation methods have the disadvantages of disrupted syntactic structures, token-label mismatch, and requirement for external knowledge or manual effort. To address these issues, we propose Robust Prompt-based Data Augmentation (RoPDA) for low-resource NER. Based on pre-trained language models (PLMs) with continuous prompt, RoPDA performs entity augmentation and context augmentation through five fundamental augmentation operations to generate label-flipping and label-preserving examples. To optimize the utilization of the augmented samples, we present two techniques: self-consistency filtering and mixup. The former effectively eliminates low-quality samples with a bidirectional mask, while the latter prevents performance degradation arising from the direct utilization of labelflipping samples. Extensive experiments on three popular benchmarks from different domains demonstrate that RoPDA significantly improves upon strong baselines, and also outperforms state-of-the-art semi-supervised learning methods when unlabeled data is included.

Equivariant message passing neural networks have emerged as the prevailing approach for predicting chemical properties of molecules due to their ability to leverage translation and rotation symmetries, resulting in a strong inductive bias. However, the equivariant operations in each layer can impose excessive constraints on the function form and network flexibility. To address these challenges, we introduce a novel network called the Hybrid Directional Graph Neural Network (HDGNN), which effectively combines strictly equivariant operations with learnable modules. We evaluate the performance of HDGNN on the QM9 dataset and the IS2RE dataset of OC20, demonstrating its state-of-the-art performance on several tasks and competitive performance on others. Our code is anonymously released on https://github.com/ajy112/HDGNN.

There has been a growing interest in using unmanned-aerial-vehicles (UAVs) for high-rate wireless communications due to their flexibility in deployment and relatively low costs. In this article, we consider an UAV-assisted wireless network, where a multiantenna ground base station provides communication services to a group of users through a set of UAVs using the in-band wireless backhaul. We propose a novel framework for optimizing the multi-UAV-assisted wireless network and consider the following two problems: one minimizes the total system cost and the other maximizes the system utility. In contrast to the previous studies, we consider the deployment cost of UAVs and the impact of the UAV locations on the backhaul capacity for both the problems. The two problems are complicated mixed-integer nonlinear programming (MINLP) problems, which involve the joint optimization of the UAV selection and their locations, the UAV-user association, and the wireless resource allocations subject to the Quality of Service requirements of users and the wireless backhaul capacity constraints. Therefore, we propose the block coordinate descent-based algorithms combined with successive convex approximation to solve the two problems. Simulation results show that the proposed methods are adaptable to the wireless backhaul constraints and outperform the other benchmark methods.

Simultaneously transmitting and reflecting reconfigurable intelligent surfaces (STAR-RISs) enables full-space manipulation of signal propagation. In this work, we consider a multi-STAR-RIS-assisted multiple-input multiple-output (MIMO) system for multi-users and investigate the impact of different STAR-RIS schemes on the system performance. We jointly optimize the beamforming matrix of the base station and the transmitting and reflecting coefficient (TRC) matrix of each STAR-RIS to maximize the sum rate of users. We propose a block coordinate descent (BCD) algorithm to optimize the beamforming matrix and the TRC matrices. We reformulate the optimization problem and optimize the beamforming matrix using the Lagrange duality method to reduce the computational complexity and then employ the constrained concave-convex procedure to tackle the optimization problem for the TRC matrices. When the energy splitting (ES) scheme is applied, our proposed method achieves a remarkable improvement of up to 17.97% in the user sum rate compared to the mode switching and the equal ES schemes in the multi-STAR-RIS-assisted system. Furthermore, when compared with systems assisted by multiple conventional RISs, our system can achieve a substantial gain of 38.99%. The improvement is even more pronounced compared to systems with a single STAR-RIS or a single conventional RIS.

In conventional recognition tasks, models are only trained to recognize learned targets, but it is usually difficult to collect training examples of all potential categories. In the testing phase, when models receive test samples from unknown classes, they mistakenly classify the samples into known classes. Open set recognition (OSR) is a more realistic recognition task, which requires the classifier to detect unknown test samples while keeping a high classification accuracy of known classes. In this paper, we study how to improve the OSR performance of deep neural networks from the perspective of representation learning. We employ supervised contrastive learning to improve the quality of feature representations, propose a new supervised contrastive learning method that enables the model to learn from soft training targets, and design an OSR framework on its basis. With the proposed method, we are able to make use of label smoothing and mixup when training deep neural networks contrastively, so as to improve both the robustness of outlier detection in OSR tasks and the accuracy in conventional classification tasks. We validate our method on multiple benchmark datasets and testing scenarios, achieving experimental results that verify the effectiveness of the proposed method.

In long-term video object tracking (VOT) tasks, most long-term trackers are modified from short-term trackers, which contain more and more machine learning modules to improve their performance. However, we empirically find that more modules do not necessarily lead to better results. In this paper, we make the long-term tracking framework simple by carefully selecting the cutting-edge trackers. Specifically, we propose a new long-term VOT framework that combines the benefits of two mainstream short-term tracking pipelines, i.e., the discriminative online tracker and the one-shot Siamese tracker, with a global re-detector awakened when the target is lost. Such a framework fully exploits existing advanced works from three complementary perspectives. Experimental results show that by exploiting the capabilities of existing methods instead of designing new neural networks, we can still achieve remarkable results on seven long-term VOT datasets. By introducing a continuous adjustable speed control parameter, our tracker reaches 20+FPS with only a small performance loss. The refine module not only improves the bounding box estimations but also outputs segmentation masks, so that our framework can handle the video object segmentation (VOS) tasks by using only VOT trackers. We obtain a trade-off between time and accuracy on two representative VOS datasets by only using bounding boxes as the initial input.

Deep neural networks are powerful, but they also have short-comings such as their sensitivity to adversarial examples, noise, blur, occlusion, etc. Moreover, ensuring the reliability and robustness of deep neural network models is crucial for their application in safety-critical areas. Much previous work has been proposed to improve specific robustness. However, we find that the specific robustness is often improved at the sacrifice of the additional robustness or generalization ability of the neural network model. In particular, adversarial training methods significantly hurt the generalization performance on unperturbed data when improving adversarial robustness. In this paper, we propose a new data processing and training method, called AugRmixAT, which can simultaneously improve the generalization ability and multiple robustness of neural network models. Finally, we validate the effectiveness of AugRmixAT on the CIFAR-10/100 and Tiny-ImageNet datasets. The experiments demonstrate that AugR-mixAT can improve the model's generalization performance while enhancing the white-box robustness, black-box robustness, common corruption robustness, and partial occlusion robustness.

Face hallucination is a task of generating high-resolution (HR) face images from low-resolution (LR) inputs, which is a subfield of the general image super-resolution. However, most of the previous methods only consider the visual effect, ignoring how to maintain the identity of the face. In this work, we propose a novel face hallucination model, called C-Face network, which can generate HR images with high visual quality while preserving the identity information. A face recognition network is used to extract the identity features in the training process. In order to make the reconstructed face images keep the identity information to a great extent, a novel metric, i.e., C-Face loss, is proposed. We also propose a new training algorithm to deal with the convergence problem. Moreover, since our work mainly focuses on the recognition accuracy of the output, we integrate face recognition into the face hallucination process which ensures that the model can be used in real scenarios. Extensive experiments on two large scale face datasets demonstrate that our C-Face network has the best performance compared with other state-of-the-art methods.

In order to quickly discover the low-dimensional representation of high-dimensional noisy data in online environments, we transform the linear dimensionality reduction problem into the problem of learning the bases of linear feature subspaces. Based on that, we propose a fast and robust dimensionality reduction framework for incremental subspace learning named evolutionary orthogonal component analysis (EOCA). By setting adaptive thresholds to automatically determine the target dimensionality, the proposed method extracts the orthogonal subspace bases of data incrementally to realize dimensionality reduction and avoids complex computations. Besides, EOCA can merge two learned subspaces that are represented by their orthonormal bases to a new one to eliminate the outlier effects, and the new subspace is proved to be unique. Extensive experiments and analysis demonstrate that EOCA is fast and achieves competitive results, especially for noisy data.

Neurobiologists recently found the brain can use sudden emerged channels to process information. Based on this finding, we put forward a question whether we can build a computation model that is able to integrate a sudden emerged new type of perceptual channel into itself in an online way. If such a computation model can be established, it will introduce a channel-free property to the computation model and meanwhile deepen our understanding about the extendibility of the brain. In this article, a biologically inspired neural network named artificial evolution (AE) network is proposed to handle the problem. When a new perceptual channel emerges, the neurons in the network can grow new connections to connect the emerged channel according to the Hebb rule. In this article, we design a sensory channel expansion experiment to test the AE network. The experimental results demonstrate that the AE network can handle the sudden emerged perceptual channels effectively.

Image clustering is more challenging than image classification. Without supervised information, current deep learning methods are difficult to be directly applied to image clustering problems. Image clustering needs to deal with three main problems: 1) the curse of dimensionality caused by highdimensional image data; 2) extracting the effective image features; 3) combining feature extraction, dimensionality reduction and clustering. In this paper, we propose a new clustering framework called Deep Embedded Dimensionality Reduction Clustering (DERC) via Probability-Based Triplet Loss, which effectively solves the above issues. To the best of our knowledge, the DERC is the first framework that effectively combines image embedding, dimensionality reduction, and clustering into the image clustering process. We also propose to incorporate a novel probability-based triplet loss measure to retrain the DERC network as a unified framework. By integrating the reconstruction loss and the probability-based triplet loss, we can improve the image clustering accuracy. Extensive experiments show that our proposed methods outperform state-of-the-art methods on many commonly used datasets.

Label distribution learning (LDL) is a newly arisen learning paradigm to deal with label ambiguity problems, which can explore the relative importance of different labels in the description of a particular instance. Although some existing LDL algorithms have achieved better effectiveness in real applications, most of them typically emphasize on improving the learning ability by manipulating the label space, while ignoring the fact that irrelevant and redundant features exist in most practical classification learning tasks, which increase not only storage requirements but also computational overheads. Furthermore, noises in data acquisition will bring negative effects on the generalization performance of LDL algorithms. In this paper, we propose a novel algorithm, i.e., Latent Semantics Encoding for Label Distribution Learning (LSE-LDL), which learns the label distribution and implements feature selection simultaneously under the guidance of latent semantics. Specifically, to alleviate noise disturbances, we seek and encode discriminative original physical/chemical features into advanced latent semantic features, and then construct a mapping from the encoded semantic space to the label space via empirical risk minimization. Empirical studies on 15 real-world data sets validate the effectiveness of the proposed algorithm.

Logs are widely used by large and complex software-intensive systems for troubleshooting. There have been a lot of studies on log-based anomaly detection. To detect the anomalies, the existing methods mainly construct a detection model using log event data extracted from historical logs. However, we find that the existing methods do not work well in practice. These methods have the close-world assumption, which assumes that the log data is stable over time and the set of distinct log events is known. However, our empirical study shows that in practice, log data often contains previously unseen log events or log sequences. The instability of log data comes from two sources: 1) the evolution of logging statements, and 2) the processing noise in log data. In this paper, we propose a new log-based anomaly detection approach, called LogRobust. LogRobust extracts semantic information of log events and represents them as semantic vectors. It then detects anomalies by utilizing an attention-based Bi-LSTM model, which has the ability to capture the contextual information in the log sequences and automatically learn the importance of different log events. In this way, LogRobust is able to identify and handle unstable log events and sequences. We have evaluated LogRobust using logs collected from the Hadoop system and an actual online service system of Microsoft. The experimental results show that the proposed approach can well address the problem of log instability and achieve accurate and robust results on real-world, ever-changing log data.

To simulate the concept acquisition and binding of different senses in the brain, a biologically inspired neural network model named perception coordination network (PCN) is proposed. It is a hierarchical structure, which is functionally divided into the primary sensory area (PSA), the primary sensory association area (SAA), and the higher order association area (HAA). The PSA contains feature neurons which respond to many elementary features, e.g., colors, shapes, syllables, and basic flavors. The SAA contains primary concept neurons which combine the elementary features in the PSA to represent unimodal concept of objects, e.g., the image of an apple, the Chinese word "[píng guǒ]" which names the apple, and the taste of the apple. The HAA contains associated neurons which connect the primary concept neurons of several PSA, e.g., connects the image, the taste, and the name of an apple. It means that the associated neurons have a multimodal response mode. Therefore, this area executes multisensory integration. PCN is an online incremental learning system, it is able to continuously acquire and bind multimodality concepts in an online way. The experimental results suggest that PCN is able to handle the multimodal concept acquisition and binding effectively.

A biologically plausible neural network model named Perception Coordination Network (PCN) is proposed for online multi-modal concept acquisition and binding. It is a hierarchical structure inspired by the structure of the brain, and functionally divided into the primary sensory area (PSA), the primary sensory association area (SAA), and the higher order association area (HAA). The PSA processes many elementary features, e.g., colors, shapes, syllables, and basic flavors, etc. The SAA combines these elementary features to represent the unimodal concept of an object, e.g., the image, name and taste of an apple, etc. The HAA connects several primary sensory association areas like a function of synaesthesia, which means associating the image, name and taste of an object. PCN is able to continuously acquire and bind multi-modal concepts in an online way. Experimental results suggest that PCN can handle the multi-modal concept acquisition and binding problem effectively.

This paper proposes a new manifold learning method called "Soinnmanifold". Traditional manifold learning method needs a lot of computation and appropriate priori parameters. This has somewhat restricted the domains in which manifold learning can potentially be applied. However, with the high-dimensional inputs, our method can generate a lowdimensional manifold in the high-dimensional space and determine the intrinsic dimension automatically. Then we will use this manifold to do dimensionality reduction quickly. Experiments demonstrate that our method can get promising results with less time and memory.

Time series classification (TSC) problem is important due to the pervasiveness of time series data. Shapelet provides a mechanism for the problem by its ability to measure local shape similarity. However, shapelets need to be searched from massive sub-sequences. To address this problem, this paper proposes a novel shapelet learning method for time series classification. The proposed method uses a self-organizing incremental neural network to learn shapelet candidates. The learned candidates reduce greatly in quantity and improve much in quality. After that, an exponential function is proposed to transform the time series data. Besides, all shapelets are selected at the same time by using an alternative attribute selection technique. Experimental results demonstrate statistically significant improvement in terms of accuracies and running speeds against 10 baselines over 28 time series datasets.

Pixel clustering is one of the basic methods for image segmentation. A critical problem of pixel clustering is how to measure the similarity between colors of the pixels on human visual perception. In this paper, we propose an adaptive clustering method for image segmentation, namely Prototypes Extraction and Merging (PEM) method. We first build a prototype network based on the Hebbian learning rule to represent the image. Then we use the Density Peaks Based Clustering (DPBC) method on the prototypes rather than the pixels for clustering in multiple color spaces, in which we choose a tiny Euclidean distance to approximate the similarity of the neighboring prototypes and find the density peaks. Finally we conduct the Multi-Space Merging (MSM) method to merge the color regions in multiple color spaces and get the final segmentation. In PEM the similarity measuring of colors is achieved adaptively according to the complexity and local contrast ratio of the image. Thus, it is extremely close to the discriminability of human visual system.

Clustering is an important technique widely used in many areas such as machine learning, pattern recognition, data analysis etc. Data stream clustering is a branch of clustering that draws much attention in recent years, where data objects are processed as an ordered sequence. In this paper, we propose an unsupervised learning neural network named Density Based Self Organizing Incremental Neural Network(DenSOINN) for data stream clustering tasks. DenSOINN is a self organizing competitive network that grows incrementally to learn suitable nodes to fit the distribution of learning data, combining on-line unsupervised learning and topology learning by means of competitive Hebbian learning rule [19]. By adopting a density-based clustering mechanism, DenSOINN can discover arbitrarily shaped clusters and diminish the negative effect of noise. In addition, we adopt a self-adaptive distance framework to obtain good performance for learning unnormalized input data. Experiments show that the DenSOINN can achieve high standard performance equally on both raw data and normalized data.

The proposed perception evolution network (PEN) is a biologically inspired neural network model for unsupervised learning and online incremental learning. It is able to automatically learn suitable prototypes from learning data in an incremental way, and it does not require the predefined prototype number or the predefined similarity threshold. Meanwhile, being more advanced than the existing unsupervised neural network model, PEN permits the emergence of a new dimension of perception in the perception field of the network. When a new dimension of perception is introduced, PEN is able to integrate the new dimensional sensory inputs with the learned prototypes, i.e., the prototypes are mapped to a high-dimensional space, which consists of both the original dimension and the new dimension of the sensory inputs. In the experiment, artificial data and real-world data are used to test the proposed PEN, and the results show that PEN can work effectively.

Manifold Learning methods aim to find meaningful low-dimensional structures hidden in their high-dimensional observations. Recently, they are faced with critical problems of how to reduce computational and space complexity in big data applications, how to determine neighborhood size adaptive to different data sets and how to deal with new observations in an out-of-sample mode. This paper presents a new method called TLOE (Topology Learning and Out-of-sample Embedding) to deal with the above three problems. TLOE uses the competitive Hebbian rule to construct the topology preserving network on a given manifold. It is capable of: 1) automatical selection of the right number and position of landmarks, 2) adaptive determination of neighborhood sizes for landmarks and 3) online embedding of new observations. Experiments on both synthetic and real-world data sets show its promising results.

The imbalance problem exists in P300 EEG data sets because P300 potential are collected under the condition of Oddball experimental paradigm. Hence, a P300 detection method, namely RUSBagging SVMs, is proposed in this paper to solve the imbalance problem and make an improvement. This algorithm re-samples the data sets at first to generate a rebalanced training set in one round of iteration and trains an SVM classifier based on the training set. Next, the SVM classifiers are integrated to make a final decision. In the integration of several classifiers, the information that is lost in the under-sampling process is generally considered. Therefore, the method is relatively robust. The experiments of character recognition based on P300 EEG data signals are conducted to examine the method. It is concluded from the experiments that RUSBagging method can indeed improve the performance of P300 detection by solving the imbalance problem in EEG data sets.

We propose a lifelong learning method for Support Vector Machine (SVM) training called L3-SVM. It focuses on the large-scale and endless stream data which are usually the characteristics of lifelong learning. The batch SVM cannot handle large-scale data for the huge space and time requirements and also cannot learn incrementally. In order to solve these two problems, we introduce a Prototype Support Layer (PSL) before SVM training, which is maintained by a Learning Prototype Network (LPN). The LPN learns representative prototypes from the input data in an online way and the PSL records the learned prototypes based on which the SVM is trained. As the size of the prototypes in the PSL is small in comparison with original data, L3-SVM is very fast. Another challenge of lifelong learning is the “open-ended” environment, i.e. data from novel classes or new distributions may occur during learning, upon which the classifiers should be retrained or updated. Many versions of incremental SVM only retain the Support Vectors after training, losing much potentially useful information of the original data, which leads to a declined performance as new data arrives. The PSL, on the other hand, works as a Long Term Memory (LTM) system. It records the representative prototypes of all previous data. Experiments demonstrate that the SVM trained upon these representative prototypes is as accurate as the state-of-the-art SVM, and much faster and can handle much larger data set than existing methods. Moreover, due to the incremental and self-adaptive properties of LPN, L3-SVM is able to work efficiently and effectively when novel class data come.

Mining time series data has been revived in the last decade due to the increasing availability of time series datasets. This paper presents an online incremental learning algorithm for time series based on the self-organizing incremental neural network (SOINN) and fast dynamic time warping (FastDTW), referred to as OILFTS. The proposed method OILFTS adopts FastDTW distance as the similarity measure, meeting the requirements of most real-time applications. Moreover, OILFTS achieves online and incremental learning of data series which are of equal or unequal length. We test our method with UCR time series datasets, and experimental results show that, from the respect of classification accuracy, the proposed OILFTS is much better than the state-of-the-art similarity measure approaches and widely investigated kernel-based SVMs.

This paper presents an extended Isomap algorithm called SL-Isomap (SOINN Landmark Isomap). We adopt SOINN (Self-Organizing Incremental Neural Network) algorithm to choose the reasonable number of landmarks automatically. SOINN landmarks are able to represent topological structure of unsupervised data in the high dimensional input space. Then L-Isomap (Landmark Isomap) algorithm is used to find low dimensional manifolds from high dimensional data based on chosen landmarks. SL-Isomap solves the problem of selecting the right number and position of landmarks automatically thus reduces short-circuit errors. It also realizes data compression and nonlinear dimensionality reduction at the same time. Experiments demonstrate its promising results compared with other variants of L-Isomap.

This paper proposes a real-time dynamic hand gesture recognition system based on Hidden Markov Models with incremental learning method (IL-HMMs) to provide natural human-computer interaction. The system is divided into four parts: hand detecting and tracking, feature extraction and vector quantization, HMMs training and hand gesture recognition, incremental learning. After quantized hand gesture vector being recognized by HMMs, incremental learning method is adopted to modify the parameters of corresponding recognized model to make itself more adaptable to the coming new gestures. Experiment results show that comparing with traditional one, the proposed system can obtain better recognition rates.

This paper presents a batch learning algorithm and an online learning algorithm for radial basis function networks based on the self-organizing incremental neural network (SOINN), together referred to as SOINN-RBF. The batch SOINN-RBF is a combination of SOINN and least square algorithm. It achieves a comparable performance with SVM for regression. The online SOINN-RBF is based on the self-adaption procedure of SOINN and adopts the growing and pruning strategy of the minimal resource allocation network (MRAN). The growing and pruning criteria use the redefined significance, which is originally introduced by the growing and pruning algorithm for RBF (GGAP-RBF). Simulation results for both artificial and real-world data sets show that, comparing with other online algorithms, the online SOINN-RBF has comparable approximation accuracy, network compactness and better learning efficiency.

In recent years, neural networks is increasingly adopted in the prediction of exchange rate. However, most of them predict a specific number, which can not help the speculators too much because small gap between the predicted values and the actual values will lead to disastrous consequences. In our study, our purpose is to present a model to forecast the fluctuation range of the exchange rate by combining Fuzzy Granulation with Continuous-valued Deep Belief Networks (CDBN), and the concept of "Stop Loss" is introduced for making the environment of our profit strategy close to the real foreign exchange trade market. The proposed model is applied to forecasting both Euro/US dollar and British pound/US dollar exchange rate in our experiments. Experimental results show that the proposed method is more profitable in the trading process than other typical models.

Global path planning as applied to mobile robotics can be approached in a similar fashion as classic optimization problems involving combinational constraints (e.g. the Traveling Salesman Problem). A single, exact optimal solution for the shortest path may not exist, and obtaining near-optimal solutions selected and ranked by criteria, or deemed “good-enough”, can satisfy the problem. An overview is provided on a select subset of naturally inspired iterative search algorithms; Simulated Annealing (SA), Genetic Algorithm (GA), and Ant Colony Optimization (ACO) have all been studied and applied to the task of mobile robotic path planning. These three techniques or algorithms (respectively) represent a broader range of naturally inspired physical processes, evolutionary or biological processes, and animal kingdom behavioral examples. It has been demonstrated that these algorithms have been utilized on their own, or as part of a collaborative hybridization of iterative algorithms and heuristic modifiers, to effectively balance the constraints, strengths and weaknesses in a given path planning approach. A brief contextual summary of current literature provides insights regarding implementation of this category of algorithms, and suggests approaches for future experimentation and research in this topic area.

Artificial neural networks (ANNs) have been proved to be successfully used in a variety of pattern recognition and data mining applications. However, training ANNs on large scale datasets are both data-intensive and computation-intensive. Therefore, large scale ANNs are used with reservation for their time-consuming training to get high precision. In this paper, we present cNeural, a customized parallel computing platform to accelerate training large scale neural networks with the backpropagation algorithm. Unlike many existing parallel neural network training systems working on thousands of training samples, cNeural is designed for fast training large scale datasets with millions of training samples. To achieve this goal, firstly, cNeural adopts HBase for large scale training dataset storage and parallel loading. Secondly, it provides a parallel in-memory computing framework for fast iterative training. Third, we choose a compact, event-driven messaging communication model instead of the heartbeat polling model for instant messaging delivery. Experimental results show that the overhead time cost by data loading and messaging communication is very low in cNeural and cNeural is around 50 times faster than the solution based on Hadoop MapReduce. It also achieves nearly linear scalability and excellent load balancing.

Selecting attention is an important cognitive psychology concept originally which has received much attention from scholars in the field of computer science. Nowadays, selecting attention has much application in computer vision. Most current computational models of attention focus on bottom-up features and ignore scene information. In this paper, a model of selecting visual attention guidance based on both bottom-up and top-down features was proposed. We used two datasets to evaluate the performance of the model, and also compare ours with Itti's model, which is particularly famous for visual attention. Experiments indicate that our model is applicable to the simulation of visual attention, and it achieves better performance in attention transferring than the models existed.

Gaussian Mixture Models has been widely used in speaker recognition during the last decades. To deal with the dynamic growth of datasets, initial clustering problem and achieving the results of clustering effectively on incremental data, an incremental adaptation method called incremental learning Gaussian mixture model (IGMM) is proposed in this paper. It was applied to speaker recognition system based on Self Organization Incremental Learning Neural Network (SOINN) and improved EM algorithm. SOINN is a Neural Network which can reach a suitable mixture number and appropriate initial cluster for each model. First, the initial training is conducted by SOINN and EM algorithm only need a limited amount of data. Then, the model would adapt to the data available in each session to enrich itself incrementally and recursively. Experiments were taken on the 1st speech separation challenge database. The results show that IGMM outperforms GMM and classical Bayesian adaptation in most of the cases.

A perception evolution network (PEN) is proposed for unsupervised fast incremental (or on-line) learning in this paper. The network has two layers: the perception layer receives the external data from the reality environment and the knowledge layer learns and records the knowledge contained in the data from the perception layer in incremental and self-organizing way. In PEN model, new input channels can be added to the perception layer freely during learning. When the perception layer gets some new input channels, the network will create corresponding data transmission channels to the knowledge layer. The prior learned knowledge stored in the knowledge layer will be expanded to a higher-dimensional space which contains the attributes of the new input channels. For incremental (or on-line) learning, PEN can automatically obtain suitable prototypes and find the topology structure of the learning data without any priori knowledge. The noise processing mechanism guarantees PEN can work in the complex real-world (noisy) environment. The experiments for both artificial data set and real-world data set show that the proposed method is effective.

This paper studies empirically the effect of different sampling methods on training classifiers on the imbalanced data of the BCI P300 Speller. Both over-sampling and under-sampling are considered. Besides some existing methods like SMOTE that have been shown to be effective in addressing the class imbalance problem we also proposed a new under-sampling technology, namely, instance-remove algorithm which is based on the property of P300 data sets. The classifiers for testing are FLDA and linear SVM. Experimental results suggest that not all of the sampling methods are effective in P300 detection, and even the same method may have different influence on different classifiers. It reveals that the SMOTE technique which is a variant of over-sampling is very effective in training an FLDA classifier while other methods are slightly effective or ineffective both in training FLDA and Linear SVM. The study also suggests that the over-sampling is more effective than under-sampling on both classifiers.

Making recognition more reliable under the condition of single sample per person is a great challenge in computer vision. In this paper, we propose a subspace based face recognition system which focuses on dealing with this problem. Inspired by the Single Image Subspace (SIS) method and the concept of typical machine learning algorithms, we design an online incremental learning system which can keep learning information from input images to improve the system performance. By combining the strengths of principal angles based similarity measure, a threshold policy and a novel sample subspace updating algorithm, the task of robust face recognition is accomplished. Experimental results on AR and EYALE database are presented to demonstrate the effectiveness of the proposed method.

Feature selection is an effective tool to deal with the "curse of dimensionality". To cope with the non-separable problem, feature selection in the kernel space has been investigated. However, previous study cannot adequately estimate the intrinsic dimensionality of the kernel space. Thus, it is difficult to accurately preserve the sketch of the kernel space using the learned basis, and the feature selection performance is affected. Moreover, the computing load of the algorithm reaches at least cubic with the number of training data. In this paper, we propose a fast framework to conduct feature selection in the kernel space. By designing a fast kernel subspace learning method, we automatically learn the intrinsic dimensionality and construct an orthogonal basis set of kernel space. The learned basis can accurately preserve the sketch of kernel space. Then backed by the constructed basis, we directly select features in kernel space. The whole proposed framework has a quadratic complexity with the number of training data, which is faster than existing kernel methods for feature selection. We evaluate our work under several typical datasets and find it not only preserves the sketch of the kernel space more accurately but also achieves better classification performance compared with many state-of-the-art methods.

Forecasting exchange rates is an important financial problem which has received much attention. Nowadays, neural network has become one of the effective tools in this research field. In this paper, we propose the use of a deep belief network (DBN) to tackle the exchange rate forecasting problem. A DBN is applied to predict both British Pound/US dollar and Indian rupee/US dollar exchange rates in our experiments. We use six evaluation criteria to evaluate its performance. We also compare our method to a feedforward neural network (FFNN), which is the state-of-the-art method for forecasting exchange rate with neural networks. Experiments indicate that deep belief networks (DBNs) are applicable to the prediction of foreign exchange rate, since they achieve better performance than feedforward neural networks (FFNNs).

Multi-instance learning, as other machine learning tasks, also suffers from the curse of dimensionality. Although dimensionality reduction methods have been investigated for many years, multi-instance dimensionality reduction methods remain untouched. On the other hand, most algorithms in multi- instance framework treat instances in each bag as independently and identically distributed samples, which fails to utilize the structure information conveyed by instances in a bag. In this paper, we propose a multi-instance dimensionality reduction method, which treats instances in each bag as non-i.i.d. samples. We regard every bag as a whole entity and define a bag margin objective function. By maximizing the margin of positive and negative bags, we learn a subspace to obtain more salient representation of original data. Experiments demonstrate the effectiveness of the proposed method.

An associative memory (AM) system is proposed to realize incremental learning and temporal sequence learning. The proposed system is constructed with three layer networks: The input layer inputs key vectors, response vectors, and the associative relation between vectors. The memory layer stores input vectors incrementally to corresponding classes. The associative layer builds associative relations between classes. The proposed method can incrementally learn key vectors and response vectors; store and recall both static information and temporal sequence information; and recall information from incomplete or noise-polluted inputs. Experiments using binary data, real-value data, and temporal sequences show that the proposed method works well.

Feature extraction is an effective tool in data mining and machine learning. Many feature extraction methods have been investigated recently. However, few methods can achieve orthogonal components. Non-orthogonal components distort the metric structure of original data space and contain reductant information. In this paper, we propose a feature extraction method, named as incremental orthogonal basis analysis (IOBA), to cope with the challenging endeavors. First, IOBA learns orthogonal components for original data, not only theoretically but also numerically. Second, an innovative way of training data selection is proposed. This selection scheme helps IOBA pick up numerically orthogonal components from training patterns. Third, by designing a self-adaptive threshold technique, no prior knowledge about the number of components is necessary to use IOBA. Moreover, without solving eigenvalue and eigenvector problems, IOBA not only saves large computing loads, but also avoids ill-conditioned problems. Results of experiments show the efficiency of the proposed IOBA.

We introduce a prototype-based nearest-neighbor method that is based on a self-organizing incremental neural network (SOINN). It automatically learns the number of prototypes necessary to determine the decision boundary, and it is robust to noisy training data. The experiments with artificial datasets and real-world datasets illustrate the efficiency of the proposed method.

An online semi-supervised active learning algorithm is proposed, which is based on self-organizing incremental neural network (SOINN). The proposed method do not need any priori knowledge such as number of nodes or number of classes; it can automatically learn number of nodes and teacher vectors needed by the current task; It can realize online incremental learning even life-long learning. The experiments for artificial data and real world data show that the proposed method works efficiently.

This paper presents an online semi-supervised clustering algorithm based on a self-organizing incremental neural network (SOINN). Using labeled data and a large amount of unlabeled data, the proposed semi-supervised SOINN (ssSOINN) can automatically learn the topology of input data distribution without any prior knowledge such as the number of nodes or a good network structure; it can subsequently divide the structure into sub-structures as the need arises. Experimental results we obtained for artificial data and real-world data show that the ssSOINN has superior performance for separating data distributions with high-density overlap and that ssSOINN Classifier (S3C) is an efficient classifier.

An on-line learning mechanism is proposed for unsupervised data. Using a similarity threshold and local error based insertion criterion, the system is able to grow incrementally and to accommodate input patterns of online non-stationary data distribution. The definition of a utility parameter -"error-radius" - enables this system to learn the number of nodes needed to solve a task. The usage of a new technique for removing nodes in low probability density regions can separate the clusters with low-density overlaps and dynamically eliminate noise in the input data. Experiment results show that this system can report a reasonable number of clusters and represent the topological structure of unsupervised on-line data with no prior conditions such as a suitable number of nodes or a good initial codebook.

Fractal image coding is a computationally expensive method. This paper does some improvement for traditional no search method and quadtree method, then combines the improved no search method with improved quadtree method to greatly speed up the fractal encoding process and hold high reconstruction fidelity. Some other time-consuming parts of fractal coding are redesigned and accelerated with new techniques. Experiments on standard images show that the proposed scheme can reduce encoding time greatly with only a little loss of approximation quality.

An on-line unsupervised learning mechanism is proposed for unlabeled data which is polluted by noises. By using a similarity threshold and local error based insertion criterion, the system is able to grow incrementally and to accommodate input patterns of on-line non-stationary data distribution. The definition of a utility parameter - "error-radius" enables this system to learn the number of nodes needed to solve a current task. The usage of a new technique for removing nodes in low probability density regions can separate the clusters with low-density overlaps and dynamically eliminate the noise in the input data. The design of two-layer neural network makes it possible for this system to represent the topological structure of unsupervised on-line data, report the reasonable number of clusters and give typical prototype patterns of every cluster without any priori conditions such as suitable number of nodes or a good initial codebook.