Dimensionality reduction (DR) is one of the key tools for the visual exploration of high-dimensional data and uncovering its cluster structure in two- or three-dimensional spaces. The vast majority of DR methods in the literature do not take into account any prior knowledge a practitioner may have regarding the dataset under consideration. We propose a novel method to generate informative embeddings which not only factor out the structure associated with different kinds of prior knowledge but also aim to reveal any remaining underlying structure. To achieve this, we employ a linear combination of two objectives: firstly, contrastive PCA that discounts the structure associated with the prior information, and secondly, kurtosis projection pursuit which ensures meaningful data separation in the obtained embeddings. We formulate this task as a manifold optimization problem and validate it empirically across a variety of datasets considering three distinct types of prior knowledge. Lastly, we provide an automated framework to perform iterative visual exploration of high-dimensional data.

The semantic matching problem consists of recognizing if the candidate text is relevant to a particular input text. Semantic similarities can be determined from human-curated knowledge, but such knowledge may not be available in every language. Instead, statistical learning techniques have been applied, but these techniques circumvent the need for manual feature engineering by using large datasets to train models to perform semantic similarity scoring between portions of text or words. The pre-trained transformer provides a further mechanism to consolidate the information throughout a sentence into single sentence-level representations, but these representations may not be optimal for the matching task. As an alternative, we propose an interactive semantic transformer based on a greedy layer-wise framework to learn a distributed similarity representation for sentence pairs. The novelty of the architecture lies in an abstract representation of the semantic similarities created by three-stage learning strategies. Model training is accomplished through a greedy layer-wise training scheme, that incorporates both supervised and unsupervised learning. The proposed model is experimentally compared to state-of-the-art approaches on three different dataset types: the library TREC, the Yahoo!, and Stack Exchange community question datasets, and results show the proposed model outperforming other approaches.

While exogenous variables have a major impact on performance improvement in time series analysis, interseries correlation and time dependence among them are rarely considered in the present continuous methods. The dynamical systems of multivariate time series could be modeled with complex unknown partial differential equations (PDEs) which play a prominent role in many disciplines of science and engineering. In this article, we propose a continuous-time model for arbitrary-step prediction to learn an unknown PDE system in multivariate time series whose governing equations are parameterized by self-attention and gated recurrent neural networks. The proposed model, exogenous-guided PDE network (EgPDE-Net), takes account of the relationships among the exogenous variables and their effects on the target series. Importantly, the model can be reduced into a regularized ordinary differential equation (ODE) problem with specially designed regularization guidance, which makes the PDE problem tractable to obtain numerical solutions and feasible to predict multiple future values of the target series at arbitrary time points. Extensive experiments demonstrate that our proposed model could achieve competitive accuracy over strong baselines: on average, it outperforms the best baseline by reducing 9.85% on RMSE and 13.98% on MAE for arbitrary-step prediction.

Kernel design is a pivotal but challenging aspect of time series analysis, especially in the context of small datasets. In recent years, Reservoir Computing (RC) has emerged as a powerful tool to compare time series based on the underlying dynamics of the generating process rather than the observed data. However, the performance of RC highly depends on the hyperparameter setting, which is hard to interpret and costly to optimize because of the recurrent nature of RC. Here, we present a new kernel for time series based on the recently established equivalence between reservoir dynamics and Nonlinear Vector AutoRegressive (NVAR) processes. The kernel is non-recurrent and depends on a small set of meaningful hyperparameters, for which we suggest an effective heuristic. We demonstrate excellent performance on a wide range of real-world classification tasks, both in terms of accuracy and speed. This further advances the understanding of RC representation learning models and extends the typical use of the NVAR framework to kernel design and representation of real-world time series data.

Deciphering oracle bone script is of great significance to the study of ancient Chinese culture as well as archaeology. Although recent studies on oracle character recognition have made substantial progress, they still suffer from the long-tailed data situation that results in a noticeable performance drop on the tail classes. To mitigate this issue, we propose a generative adversarial framework to augment oracle characters in the problematic classes. In this framework, the generator produces synthetic data through convex combinations of all the available samples in the corresponding classes, and is further optimized through adversarial learning with the classifier and simultaneously the discriminator . Meanwhile, we introduce Repatch to generalize samples in the generator. Since tail classes do not have sufficient data for convex combinations , we propose the TailMix mechanism to generate suitable tail class samples from other classes. Experimental results show that our proposed algorithm obtains remarkable performance in oracle character recognition and achieves new state-of-the-art average (total) accuracy with 86.03% (89.46%), 86.54% (93.86%), 95.22% (96.17%) on the three datasets Oracle-AYNU, OBC306 and Oracle-20K, respectively.

In this work, we propose a continuous neural network architecture, referred to as Explainable Tensorized Neural - Ordinary Differential Equations (ETN-ODE) network for multi-step time series prediction at arbitrary time points. Unlike existing approaches which mainly handle univariate time series for multi-step prediction, or multivariate time series for single-step predictions, ETN-ODE is capable of handling multivariate time series with arbitrary-step predictions. An additional benefit is its tandem attention mechanism, with respect to temporal and variable attention, which enable it to greatly facilitate data interpretability. Specifically, the proposed model combines an explainable tensorized gated recurrent unit with ordinary differential equations, with the derivatives of the latent states parameterized through a neural network. We quantitatively and qualitatively demonstrate the effectiveness and interpretability of ETN-ODE on one arbitrary-step prediction task and five standard multi-step prediction tasks. Extensive experiments show that the proposed method achieves very accurate predictions at arbitrary time points while attaining very competitive performance against the baseline methods in standard multi-step time series prediction.

The question answering system in open domain enables a machine to automatically select and generate the answer for questions posed by humans in a natural language form on the website. Previous approaches seek effective ways of extracting the semantic features between question and answer, but the contextual information effects in semantic matching are still limited by short-term memory. As an alternative, we propose an internal knowledge-based end-to-end model, enhanced by an attentive memory network for both answer selection and answer generation tasks by considering the full advantages of the semantics and multifacts (i.e., timescales, topics, and context). In detail, we design a long-term memory to learn the top-$k$fine-grained similarity representations, where two memory-aware mechanisms aggregate the series of semantic word-level and sentence-level similarities to support the coarse contextual information. Furthermore, we propose a novel memory refinement mechanism with the two-dimensional of writing heads that offer an efficient approach to multiview selection of the salient word pairs. In the training stage, we adopt the transformer-based transfer learning skill to effectively pretrain the model. Experimentally, we compare the state-of-the-art approaches on four public datasets, the experimental results show that the proposed model achieves competitive performance.

The type of neural networks widely used in artificial intelligence applications mixes its computation and memory modules in neuron weights and activities. The previously learned information are stored in network weights. When dealing with complex data, e.g., those possessing diverse content or containing long-sequences, some information stored in the weights can be altered drastically or wiped as the training goes, but they are not necessarily unimportant. External memory is a recent technique proposed to prevent from forgetting significant previously learned information. In this work, we aim at taking advantage of this recent technique to advance the similarity learning task that is critical in many real-world artificial intelligence applications. We propose suitable external memory design supported by extended attention mechanism. Two different kinds of memory modules are proposed so that the similarity learning process can dynamically shift focus over a wide range of diverse content contained by the training data. Effectiveness of the proposed method is demonstrated through evaluations based on different image retrieval tasks and compared against various state-of-the-art algorithms in the field.

This paper details FPGA implementation methodology for Convolutional Spiking Neural Networks (CSNN) and applies this methodology to low-power radioisotope identification using high resolution data. A power consumption of 75 mW has been achieved on an FPGA implementation of a CSNN, with the inference accuracy of 90.62% on a synthetic dataset. The chip validation method is presented. Prototyping was accelerated by evaluating SNN parameters using SpiNNaker neuromorphic platform.

In this paper, we are tackling the weakly-supervised referring expression grounding task, for the localization of a referent object in an image according to a query sentence, where the mapping between image regions and queries are not available during the training stage. In traditional methods, an object region that best matches the referring expression is picked out, and then the query sentence is reconstructed from the selected region, where the reconstruction difference serves as the loss for back-propagation. The existing methods, however, conduct both the matching and the reconstruction approximately as they ignore the fact that the matching correctness is unknown. To overcome this limitation, a discriminative triad is designed here as the basis to the solution, through which a query can be converted into one or multiple discriminative triads in a very scalable way. Based on the discriminative triad, we further propose the triad-level matching and reconstruction modules which are lightweight yet effective for the weakly-supervised training, making it three times lighter and faster than the previous state-of-the-art methods. One important merit of our work is its superior performance despite the simple and neat design. Specifically, the proposed method achieves a new state-of-the-art accuracy when evaluated on RefCOCO (39.21 percent), RefCOCO+ (39.18 percent) and RefCOCOg (43.24 percent) datasets, that is 4.17, 4.08 and 7.8 percent higher than the previous one, respectively. The code is available at https://github.com/insomnia94/DTWREG.

In general, development of adequately complex mathematical models, such as deep neural networks, can be an effective way to improve the accuracy of learning models. However, this is achieved at the cost of reduced post-hoc model interpretability, because what is learned by the model can become less intelligible and tractable to humans as the model complexity increases. In this paper, we target a similarity learning task in the context of image retrieval, with a focus on the model interpretability issue. An effective similarity neural network (SNN) is proposed to offer not only to seek robust retrieval performance but also to achieve satisfactory post-hoc interpretability. The network is designed by linking the neuron architecture with the organization of a concept tree and by formulating neuron operations to pass similarity information between concepts. Various ways of understanding and visualizing what is learned by the SNN neurons are proposed. We also exhaustively evaluate the proposed approach using a number of relevant datasets against a number of state-of-the-art approaches to demonstrate the effectiveness of the proposed network. Our results show that the proposed approach can offer superior performance when compared against state-of-the-art approaches. Neuron visualization results are demonstrated to support the understanding of the trained neurons.

Sentiment analysis is an important topic concerning identification of feelings, attitudes, emotions and opinions from text. To automate such analysis, a large amount of example text needs to be manually annotated for model training. This is laborious and expensive, but the cross-domain technique is a key solution to reducing the cost by reusing annotated reviews across domains. However, its success largely relies on the learning of a robust common representation space across domains. In the recent years, significant effort has been invested to improve the cross-domain representation learning by designing increasingly more complex and elaborate model inputs and architectures. We support that it is not necessary to increase design complexity as this inevitably consumes more time in model training. Instead, we propose to explore the word polarity and occurrence information through a simple mapping and encode such information more accurately whilst managing lower computational costs. The proposed approach is unique and takes advantage of the stochastic embedding technique to tackle cross-domain sentiment alignment. Its effectiveness is benchmarked with over ten data tasks constructed from two review corpora and it is compared against ten classical and state-of-the-art methods.

Recent prevalent one stage detectors, such as single shot detector (SSD) and RetinaNet, are able to detect objects faster than two stage ones while maintaining comparable accuracy. To further boost the accuracy, many studies focus on enhancing the multi-scale feature pyramid. Most of these current proposals focus on strengthening features on one pyramid, ignoring the rich connection among different scale features. In contrast, we propose a novel multi-path design to fully utilize the localization and semantics information. First, we exploit the original SSD multi-scale features as our base pyramid. Then we fuse these features in different groups to generate multi-path feature pyramids. Finally, we combine these pyramids through a novel and effective aggregation module, to obtain the final informative pyramid for detection. Comparative experiments on benchmark PASCAL VOC and MS COCO datasets have shown that our proposed method outperforms many state-of-the-art detectors. As an illustrative example, for input image with size 512×512, we can achieve a mean Average Precision (mAP) of 81.8% on VOC2007 test and 33.1% mAP on COCO test-dev2015.

We propose a set of highly scalable algorithms for the combinatorial data analysis problem of seriating similarity matrices. Seriation consists of finding a permutation of data instances, such that similar instances are nearby in the ordering. Applications of the seriation problem can be found in various disciplines such as in bioinformatics for genome sequencing, data visualization and exploratory data analysis. Our algorithms attempt to minimize certain p-SUM objectives, which also arise in the problem of envelope reduction of sparse matrices. In particular, we present a set of graduated non-convexity algorithms for vector-based relaxations of the general p-SUM problem for $$p \in \left\{ 2, 1, \tfrac{1}{2}\right\} $$ that can scale to very large problem sizes. Different choices of p emphasize global versus local similarity pattern structure. We conduct a number of experiments to compare our algorithms to various state-of-the-art combinatorial optimization methods on real and synthetic datasets. The experimental results demonstrate that compared to other approaches, the proposed algorithms are very competitive and scale well with large problem sizes.

Manifold embedding techniques have properties that render them attractive candidates to learn a compact and general representation of a three dimensional spatial object. In turn this representation can be used for object recognition through classification. This paper presents a comparative study of several supervised spectral embedding techniques and their relationship with the feature space used to describe the exemplars which act as inputs to an embedding procedure. By concentrating on this aspect, we are able to highlight preferential combinations between feature description and embedding, and we formulate recommendations on the use of such methods for fusing multiple views of an object to recognize it under variable poses.

A typical objective of data visualization is to generate low-dimensional plots that maximally convey the information within the data. The visualization output should help the user not only identify the local neighborhood structure of individual samples, but also obtain a global view of the relative positioning and separation between cohorts. Here, we propose a novel visualization framework designed to satisfy these needs. By incorporating additional cohort positioning and discriminative constraints into local neighbor preservation models through the use of computed cohort prototypes, effective control over the arrangements and proximities of data cohorts can be obtained. We introduce various embedding and projection algorithms based on objective functions addressing the different visualization requirements. Their underlying models are optimized effectively using matrix manifold procedures to incorporate the problem constraints. Additionally, to facilitate large-scale applications, a matrix decomposition based model is also proposed to accelerate the computation. The improved capabilities of the new methods are demonstrated using various state-of-the-art dimensionality reduction algorithms. We present many qualitative and quantitative comparisons, on both synthetic problems and real-world tasks of complex text and image data, that show notable improvements over existing techniques.

Descriptive clustering consists of automatically organizing data instances into clusters and generating a descriptive summary for each cluster. The description should inform a user about the contents of each cluster without further examination of the specific instances, enabling a user to rapidly scan for relevant clusters. Selection of descriptions often relies on heuristic criteria. We model descriptive clustering as an auto-encoder network that predicts features from cluster assignments and predicts cluster assignments from a subset of features. The subset of features used for predicting a cluster serves as its description. For text documents, the occurrence or count of words, phrases, or other attributes provides a sparse feature representation with interpretable feature labels. In the proposed network, cluster predictions are made using logistic regression models, and feature predictions rely on logistic or multinomial regression models. Optimizing these models leads to a completely self-tuned descriptive clustering approach that automatically selects the number of clusters and the number of features for each cluster. We applied the methodology to a variety of short text documents and showed that the selected clustering, as evidenced by the selected feature subsets, are associated with a meaningful topical organization.

Motoki Sato, Austin J. Brockmeier, Georgios Kontonatsios, Tingting Mu, John Y. Goulermas, Jun’ichi Tsujii, Sophia Ananiadou. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers. 2017.

In this work we propose a highly scalable algorithm for solving the combinatorial data analysis problem of seriation. Seriation is a technique for optimizing a permutation of data instances, with respect to some proximity measure such that nearby instances in the linear arrangement are more similar. One consistent objective function for seriation is the 2-SUM minimization problem, which uses the 2-norm between instance locations to penalize non-zero similarity values, and can be written as a quadratic function of the permutation vector. Recently, two convex relaxations of the 2-SUM problem have been proposed, which can be solved as constrained quadratic programs using interior point methods; however, the interior point solvers become expensive when the problem size increases. In this paper we present a graduated non-convexity method for vector-based relaxations of the 2-SUM that yields better approximate solutions and scales to very large problem sizes. We conduct a number of experiments on real and synthetic datasets. The experimental results demonstrate that our proposed algorithm outperforms other approaches that solve the 2-SUM, and is the only competitive approach that can scale to large problem sizes.

Citation screening, an integral process within systematic reviews that identifies citations relevant to the underlying research question, is a time-consuming and resource-intensive task. During the screening task, analysts manually assign a label to each citation, to designate whether a citation is eligible for inclusion in the review. Recently, several studies have explored the use of active learning in text classification to reduce the human workload involved in the screening task. However, existing approaches require a significant amount of manually labelled citations for the text classification to achieve a robust performance. In this paper, we propose a semi-supervised method that identifies relevant citations as early as possible in the screening process by exploiting the pairwise similarities between labelled and unlabelled citations to improve the classification performance without additional manual labelling effort. Our approach is based on the hypothesis that similar citations share the same label (e.g., if one citation should be included, then other similar citations should be included also). To calculate the similarity between labelled and unlabelled citations we investigate two different feature spaces, namely a bag-of-words and a spectral embedding based on the bag-of-words. The semi-supervised method propagates the classification codes of manually labelled citations to neighbouring unlabelled citations in the feature space. The automatically labelled citations are combined with the manually labelled citations to form an augmented training set. For evaluation purposes, we apply our method to reviews from clinical and public health. The results show that our semi-supervised method with label propagation achieves statistically significant improvements over two state-of-the-art active learning approaches across both clinical and public health reviews.

In this paper, we describe an unsupervised measure for quantifying the 'informativeness' of correlation matrices formed from the pairwise similarities or relationships among data instances. The measure quantifies the heterogeneity of the correlations and is defined as the distance between a correlation matrix and the nearest correlation matrix with constant off-diagonal entries. This non-parametric notion generalizes existing test statistics for equality of correlation coefficients by allowing for alternative distance metrics, such as the Bures and other distances from quantum information theory. For several distance and dissimilarity metrics, we derive closed-form expressions of informativeness, which can be applied as objective functions for machine learning applications. Empirically, we demonstrate that informativeness is a useful criterion for selecting kernel parameters, choosing the dimension for kernel-based nonlinear dimensionality reduction, and identifying structured graphs. We also consider the problem of finding a maximally informative correlation matrix around a target matrix, and explore parameterizing the optimization in terms of the coordinates of the sample or through a lower-dimensional embedding. In the latter case, we find that maximizing the Bures-based informativeness measure, which is maximal for centered rank-1 correlation matrices, is equivalent to minimizing a specific matrix norm, and present an algorithm to solve the minimization problem using the norm's proximal operator. The proposed correlation denoising algorithm consistently improves spectral clustering. Overall, we find informativeness to be a novel and useful criterion for identifying non-trivial correlation structure.

In this paper, a novel unsupervised hashing algorithm, referred to as t-USMVH, and its extension to unsupervised deep hashing, referred to as t-UDH, are proposed to support large-scale video-to-video retrieval. To improve robustness of the unsupervised learning, the t-USMVH combines multiple types of feature representations and effectively fuses them by examining a continuous relevance score based on a Gaussian estimation over pairwise distances, and also a discrete neighbor score based on the cardinality of reciprocal neighbors. To reduce sensitivity to scale changes for mapping objects that are far apart from each other, Student t-distribution is used to estimate the similarity between the relaxed hash code vectors for keyframes. This results in more accurate preservation of the desired unsupervised similarity structure in the hash code space. By adapting the corresponding optimization objective and constructing the hash mapping function via a deep neural network, we develop a robust unsupervised training strategy for a deep hashing network. The efficiency and effectiveness of the proposed methods are evaluated on two public video collections via comparisons against multiple classical and the state-of-the-art methods.

Near-duplicate video retrieval (NDVR) has been a significant research task in multimedia given its high impact in applications, such as video search, recommendation, and copyright protection. In addition to accurate retrieval performance, the exponential growth of online videos has imposed heavy demands on the efficiency and scalability of the existing systems. Aiming at improving both the retrieval accuracy and speed, we propose a novel stochastic multiview hashing algorithm to facilitate the construction of a large-scale NDVR system. Reliable mapping functions, which convert multiple types of keyframe features, enhanced by auxiliary information such as video-keyframe association and ground truth relevance to binary hash code strings, are learned by maximizing a mixture of the generalized retrieval precision and recall scores. A composite Kullback-Leibler divergence measure is used to approximate the retrieval scores, which aligns stochastically the neighborhood structures between the original feature and the relaxed hash code spaces. The efficiency and effectiveness of the proposed method are examined using two public near-duplicate video collections and are compared against various classical and state-of-the-art NDVR systems.

Descriptive document clustering aims at discovering clusters of semantically interrelated documents together with meaningful labels to summarize the content of each document cluster. In this work, we propose a novel descriptive clustering framework, referred to as CEDL. It relies on the formulation and generation of 2 types of heterogeneous objects, which correspond to documents and candidate phrases, using multilevel similarity information. CEDL is composed of 5 main processing stages. First, it simultaneously maps the documents and candidate phrases into a common co‐embedded space that preserves higher‐order, neighbor‐based proximities between the combined sets of documents and phrases. Then, it discovers an approximate cluster structure of documents in the common space. The third stage extracts promising topic phrases by constructing a discriminant model where documents along with their cluster memberships are used as training instances. Subsequently, the final cluster labels are selected from the topic phrases using a ranking scheme using multiple scores based on the extracted co‐embedding information and the discriminant output. The final stage polishes the initial clusters to reduce noise and accommodate the multitopic nature of documents. The effectiveness and competitiveness of CEDL is demonstrated qualitatively and quantitatively with experiments using document databases from different application fields.

This work is related to the combinatorial data analysis problem of seriation used for data visualization and exploratory analysis. Seriation re-sequences the data, so that more similar samples or objects appear closer together, whereas dissimilar ones are further apart. Despite the large number of current algorithms to realize such re-sequencing, there has not been a systematic way for analyzing the resulting sequences, comparing them, or fusing them to obtain a single unifying one. We propose a new positional proximity measure that evaluates the similarity of two arbitrary sequences based on their agreement on pairwise positional information of the sequenced objects. Furthermore, we present various statistical properties of this measure as well as its normalized version modeled as an instance of the generalized correlation coefficient. Based on this measure, we define a new procedure for consensus seriation that fuses multiple arbitrary sequences based on a quadratic assignment problem formulation and an efficient way of approximating its solution. We also derive theoretical links with other permutation distance functions and present their associated combinatorial optimization forms for consensus tasks. The utility of the proposed contributions is demonstrated through the comparison and fusion of multiple seriation algorithms we have implemented, using many real-world datasets from different application domains.

Unsupervised Cross-domain Sentiment Classification is the task of adapting a sentiment classifier trained on a particular domain (source domain), to a different domain (target domain), without requiring any labeled data for the target domain. By adapting an existing sentiment classifier to previously unseen target domains, we can avoid the cost for manual data annotation for the target domain. We model this problem as embedding learning, and construct three objective functions that capture: (a) distributional properties ofpivots(i.e., common features that appear in both source and target domains), (b) label constraints in the source domain documents, and (c) geometric properties in the unlabeled documents in both source and target domains. Unlike prior proposals that first learn a lower-dimensional embedding independent of the source domain sentiment labels, and next a sentiment classifier in this embedding, our joint optimisation method learns embeddings that are sensitive to sentiment classification. Experimental results on a benchmark dataset show that by jointly optimising the three objectives we can obtain better performances in comparison to optimising each objective function separately, thereby demonstrating the importance of task-specific embedding learning for cross-domain sentiment classification. Among the individual objective functions, the best performance is obtained by (c). Moreover, the proposed method reports cross-domain sentiment classification accuracies that are statistically comparable to the current state-of-the-art embedding learning methods for cross-domain sentiment classification.

Dimensionality Reduction (DR) is a fundamental yet active research topic in pattern recognition and machine learning. When used in classification, previous research usually performs DR separately, and then inputs the reduced features to other available models, e.g., Gaussian Mixture Model (GMM). Such independent learning could however significantly limit the classification performance, since the optimal subspace given by a particular DR approach may not be appropriate for the following classification model. More seriously, for high-dimensional data classification in the face of a limited number of samples (called small sample size or S3 problem), independent learning of DR and classification model may even deteriorate the classification accuracy. To solve this problem, we propose a joint learning model, called Two-layer Mixture of Factor Analyzers with Joint Factor Loading (2L-MJFA) for classification. More specifically, our proposed model enjoys a two-layer mixture structure, or a mixture of mixtures structure, with each component (representing each specific class) as another mixture model of Factor Analyzer (MFA). Importantly, all the involved factor analyzers are intentionally designed to share the same loading matrix. On one hand, such joint loading matrix can be considered as the dimensionality reduction matrix; on the other hand, a joint common matrix would largely reduce the parameters, making the proposed algorithm very suitable for S3 problems. We describe our model definition and propose a modified EM algorithm to optimize the model. A series of experiments demonstrates that our proposed model significantly outperforms the other three competitive algorithms on five data sets.

This paper proposes a novel manifold embedding method for the automated processing of large varied datasets. The method is based on binary classification, where the embeddings are constructed so as to determine one or more unique features for each class individually from a given dataset. The proposed method is applied to examples of multiclass classification that are relevant for large-scale data processing for surveillance (e.g., face recognition), where the aim is to augment decision making by reducing extremely large sets of data to a manageable level before displaying the selected subset of data to a human operator. The method consists of two stages: Preprocessing and embedding computation. In the embedding computation, adaptive measures of intraclass and interclass information are proposed, based on the concepts of “friend closeness” and “enemy dispersion.” In addition, an indicator for weighted pairwise constraint is proposed to balance the contributions from different classes to the final optimization, in order to better control the relative positions between the important data samples from either the same class (intraclass) or different classes (interclass). The effectiveness of the proposed method is evaluated through comparison with seven existing techniques for embedding learning, using four established databases of faces, consisting of various poses, lighting conditions, and facial expressions, as well as two standard text datasets. The proposed method performs better than these existing techniques, especially for cases with small sets of training data samples.

This paper proposes a novel way for generating reliable low-dimensional features with improved class separability in a kernel-induced feature space. The feature projections rely on a very efficient sequential projection pursuit method, adapted to support nonlinear projections using a new kernel matrix update scheme. This enables the gradual removal of structure from the space of residual dimensions to allow the recovery of multiple projections. An adaptive kernel function is employed to unfold different types of data characteristics. We follow a holistic model selection procedure that, together with the optimal projections, dimensionality, and kernel parameters, additionally optimizes symbolically the projection index that controls the actual measurement of the data interestingness without user interaction. We tackle the underlying complex bi-level optimization model as a mixture of evolutionary and gradient search. The effectiveness of the proposed algorithm over existing approaches is demonstrated with benchmark evaluations and comparisons.

In this paper, we study the co-embedding problem of how to map different types of patterns into one common low-dimensional space, given only the associations (relation values) between samples. We conduct a generic analysis to discover the commonalities between existing co-embedding algorithms and indirectly related approaches and investigate possible factors controlling the shapes and distributions of the co-embeddings. The primary contribution of this work is a novel method for computing co-embeddings, termed the automatic co-embedding with adaptive shaping (ACAS) algorithm, based on an efficient transformation of the co-embedding problem. Its advantages include flexible model adaptation to the given data, an economical set of model variables leading to a parametric co-embedding formulation, and a robust model fitting criterion for model optimization based on a quantization procedure. The secondary contribution of this work is the introduction of a set of generic schemes for the qualitative analysis and quantitative assessment of the output of co-embedding algorithms, using existing labeled benchmark datasets. Experiments with synthetic and real-world datasets show that the proposed algorithm is very competitive compared to existing ones.

Spectral embedding methods have played a very important role in dimensionality reduction and feature generation in machine learning. Supervised spectral embedding methods additionally improve the classification of labeled data, using proximity information that considers both features and class labels. However, these calculate the proximity information by treating all intraclass similarities homogeneously for all classes, and similarly for all interclass samples. In this paper, we propose a very novel and generic method which can treat all the intra- and interclass sample similarities heterogeneously by potentially using a different proximity function for each class and each class pair. To handle the complexity of selecting these functions, we employ evolutionary programming as an automated powerful formula induction engine. In addition, for computational efficiency and expressive power, we use a compact matrix tree representation equipped with a broad set of functions that can build most currently used similarity functions as well as new ones. Model selection is data driven, because the entire model is symbolically instantiated using only problem training data, and no user-selected functions or parameters are required. We perform thorough comparative experimentations with multiple classification datasets and many existing state-of-the-art embedding methods, which show that the proposed algorithm is very competitive in terms of classification accuracy and generalization ability.

This paper is about supervised and semi-supervised dimensionality reduction (DR) by generating spectral embeddings from multi-output data based on the pairwise proximity information. Two flexible and generic frameworks are proposed to achieve supervised DR (SDR) for multilabel classification. One is able to extend any existing single-label SDR to multilabel via sample duplication, referred to as MESD. The other is a multilabel design framework that tackles the SDR problem by computing weight (proximity) matrices based on simultaneous feature and label information, referred to as MOPE, as a generalization of many current techniques. A diverse set of different schemes for label-based proximity calculation, as well as a mechanism for combining label-based and feature-based weight information by considering information importance and prioritization, are proposed for MOPE. Additionally, we summarize many current spectral methods for unsupervised DR (UDR), single/multilabel SDR, and semi-supervised DR (SSDR) and express them under a common template representation as a general guide to researchers in the field. We also propose a general framework for achieving SSDR by combining existing SDR and UDR models, and also a procedure of reducing the computational cost via learning with a target set of relation features. The effectiveness of our proposed methodologies is demonstrated with experiments with document collections for multilabel text categorization from the natural language processing domain.

The objective of this paper is the design of an engine for the automatic generation of supervised manifold embedding models. It proposes a modular and adaptive data embedding framework for classification, referred to as DEFC, which realizes in different stages including initial data preprocessing, relation feature generation and embedding computation. For the computation of embeddings, the concepts of friend closeness and enemy dispersion are introduced, to better control at local level the relative positions of the intraclass and interclass data samples. These are shown to be general cases of the global information setup utilized in the Fisher criterion, and are employed for the construction of different optimization templates to drive the DEFC model generation. For model identification, we use a simple but effective bilevel evolutionary optimization, which searches for the optimal model and its best model parameters. The effectiveness of DEFC is demonstrated with experiments using noisy synthetic datasets possessing nonlinear distributions and real-world datasets from different application fields.

The operation of instance-based learning algorithms is based on storing a large set of prototypes in the system's database. However, such systems often experience issues with storage requirements, sensitivity to noise, and computational complexity, which result in high search and response times. In this brief, we introduce a novel framework that employs spectral graph theory to efficiently partition the dataset to border and internal instances. This is achieved by using a diverse set of border-discriminating features that capture the local friend and enemy profiles of the samples. The fused information from these features is then used via graph-cut modeling approach to generate the final dataset partitions of border and nonborder samples. The proposed method is referred to as the spectral instance reduction (SIR) algorithm. Experiments with a large number of datasets show that SIR performs competitively compared to many other reduction algorithms, in terms of both objectives of classification accuracy and data condensation.

This paper proposes a generalized locally weighted group method of data handling (G-LWGMDH) based on evolutionary algorithm (EA) for short-term load forecasting. The locally weighted group method of data handling (LWGMDH) can be derived by combining GMDH with the local regression method and weighted least squares (WLS) regression. The connectivity configuration in the G-LWGMDH is not limited to adjacent layers, unlike the conventional GMDH. Moreover, each node in the G-LWGMDH network has a different number of inputs and a different polynomial order. The performance of the G-LWGMDH depends on choosing these factors before the network is constructed. Therefore, EA is used in this paper to optimally select these factors. In the proposed method, a new encoding scheme is presented, where each chromosome represents the structure of the whole network. The weighting functions bandwidth, the polynomial order for each node, the number of inputs for each node, and the input variables chosen to each node are encoded as a chromosome. The performance of the proposed method (EA-based G-LWGMDH) is evaluated using two real-world datasets. The results show that the proposed method provides a much better prediction performance in comparison with other methods employing the same data.

Plantar lesions induced by biomechanical dysfunction pose a considerable socioeconomic health care challenge, and failure to detect lesions early can have significant effects on patient prognoses. Most of the previous works on plantar lesion identification employed the analysis of biomechanical microenvironment variables like pressure and thermal fields. This paper focuses on foot kinematics and applies kernel principal component analysis (KPCA) for nonlinear dimensionality reduction of features, followed by Fisher's linear discriminant analysis for the classification of patients with different types of foot lesions, in order to establish an association between foot motion and lesion formation. Performance comparisons are made using leave-one-out cross-validation. Results show that the proposed method can lead to approximately 94% correct classification rates, with a reduction of feature dimensionality from 2100 to 46, without any manual preprocessing or elaborate feature extraction methods. The results imply that foot kinematics contain information that is highly relevant to pathology classification and also that the nonlinear KPCA approach has considerable power in unraveling abstract biomechanical features into a relatively low-dimensional pathology-relevant space.

Projection techniques are frequently used as the principal means for the implementation of feature extraction and dimensionality reduction for machine learning applications. A well established and broad class of such projection techniques is the projection pursuit (PP). Its core design parameter is a projection index, which is the driving force in obtaining the transformation function via optimization, and represents in an explicit or implicit way the user's perception of the useful information contained within the datasets. This paper seeks to address the problem related to the design of PP index functions for the linear feature extraction case. We achieve this using an evolutionary search framework, capable of building new indices to fit the properties of the available datasets. The high expressive power of this framework is sustained by a rich set of function primitives. The performance of several PP indices previously proposed by human experts is compared with these automatically generated indices for the task of classification, and results show a decrease in the classification errors.

The forecasting of electricity demand has become one of the major research fields in electrical engineering. Accurately estimated forecasts are essential part of an efficient power system planning and operation. In this paper, a modified version of the support vector regression (SVR) is presented to solve the load forecasting problem. The proposed model is derived by modifying the risk function of the SVR algorithm with the use of locally weighted regression (LWR) while keeping the regularization term in its original form. In addition, the weighted distance algorithm based on the Mahalanobis distance for optimizing the weighting function's bandwidth is proposed to improve the accuracy of the algorithm. The performance of the new model is evaluated with two real-world datasets, and compared with the local SVR and some published models using the same datasets. The results show that the proposed model exhibits superior performance compare to that of LWR, local SVR, and other published models.

Network utility maximization (NUM) provides an important perspective to conduct rate allocation where optimal performance, in terms of maximal aggregate bandwidth utility, is generally achieved such that each source adaptively adjusts its transmission rate. Behind most of the recent literature on NUM, common assumptions are that traffic flows are elastic and that their utility functions are strictly concave. This provides design simplicity but, in practice, limits the applicability of resulting protocols, in that severe QoS problems may be encountered when bandwidth is shared by inelastic flows. This paper investigates the problem of distributively allocating data transmission rates to multiclass services, both elastic and inelastic, and overcomes the restrictive and often unrealistic assumptions. The proposed method is based on the Lagrangian Relaxation for a dual formulation that decomposes the higher dimension NUM into a number of subproblems. We use a novel Surrogate Subgradient based stochastic method to solve the dual problem. Unlike the ordinary subgradient methods, surrogate subgradient can compute optimal prices without the need to solve all the subproblems. For the lower dimension, nonlinear and nonconvex subproblems we use a hybrid particle swarm optimization (PSO) and sequential quadratic programming (SQP) method, where the objective is to achieve fast convergence as well as accuracy. We demonstrate the efficiency of the proposed rate allocation algorithm, in terms maintaining QoS for multiclass services, and validate its scalability and accuracy for large scale flows.

This paper investigates the capabilities of hierarchical fuzzy systems to approximate functions on discrete input spaces. First, it is shown that any function on a discrete space has an arbitrary separable hierarchical structure and can be naturally approximated by hierarchical fuzzy systems. As a by-product of this result, a discrete version of Kolmogorov's theorem is obtained; second, it is proven that any function on a discrete space can be approximated to any degree of accuracy by hierarchical fuzzy systems with any desired separable hierarchical structure. That is, functions on discrete spaces can be approximated more simply and flexibly than those on continuous spaces; third, a hierarchical fuzzy system identification method is proposed in which human knowledge and numerical data are combined for system construction and identification. Finally, the proposed method is applied to the market condition performance modeling problem in site selection decision support and shows the better performance in both accuracy and interpretability than the regression and neural network approaches. In additions, the reason and mechanism why hierarchical fuzzy systems outperform regression and neural networks in this type of application are analyzed.

Wearable human movement measurement systems are increasingly popular as a means of capturing human movement data in real-world situations. Previous work has attempted to estimate segment kinematics during walking from foot acceleration and angular velocity data. In this paper, we propose a novel neural network [GRNN with Auxiliary Similarity Information (GASI)] that estimates joint kinematics by taking account of proximity and gait trajectory slope information through adaptive weighting. Furthermore, multiple kernel bandwidth parameters are used that can adapt to the local data density. To demonstrate the value of the GASI algorithm, hip, knee, and ankle joint motions are estimated from acceleration and angular velocity data for the foot and shank, collected using commercially available wearable sensors. Reference hip, knee, and ankle kinematic data were obtained using externally mounted reflective markers and infrared cameras for subjects while they walked at different speeds. The results provide further evidence that a neural net approach to the estimation of joint kinematics is feasible and shows promise, but other practical issues must be addressed before this approach is mature enough for clinical implementation. Furthermore, they demonstrate the utility of the new GASI algorithm for making estimates from continuous periodic data that include noise and a significant level of variability.

This paper proposes a new algorithm for solving mixed discrete nonlinear programming (MDNLP) problems, designed to efficiently combine particle swarm optimization (PSO), which is a well-known global optimization technique, and branch-and-bound (BB), which is a widely used systematic deterministic algorithm for solving discrete problems. The proposed algorithm combines the global but slow search of PSO with the rapid but local search capabilities of BB, to simultaneously achieve an improved optimization accuracy and a reduced requirement for computational resources. It is capable of handling arbitrary continuous and discrete constraints without the use of a penalty function, which is frequently cumbersome to parameterize. At the same time, it maintains a simple, generic, and easy-to-implement architecture, and it is based on the sequential quadratic programming for solving the NLP subproblems in the BB tree. The performance of the new hybrid PSO-BB architecture algorithm is evaluated against real-world MDNLP benchmark problems, and it is found to be highly competitive compared with existing algorithms.

This paper proposes a new nonparametric regression method, based on the combination of generalized regression neural networks (GRNNs), density-dependent multiple kernel bandwidths, and regularization. The presented model is generic and substitutes the very large number of bandwidths with a much smaller number of trainable weights that control the regression model. It depends on sets of extracted data density features which reflect the density properties and distribution irregularities of the training data sets. We provide an efficient initialization scheme and a second-order algorithm to train the model, as well as an overfitting control mechanism based on Bayesian regularization. Numerical results show that the proposed network manages to reduce significantly the computational demands of having individual bandwidths, while at the same time, provides competitive function approximation accuracy in relation to existing methods.

This paper proposes a novel algorithm for function approximation that extends the standard generalized regression neural network. Instead of a single bandwidth for all the kernels, we employ a multiple-bandwidth configuration. However, unlike previous works that use clustering of the training data for the reduction of the number of bandwidths, we propose a distinct scheme that manages a dramatic bandwidth reduction while preserving the required model complexity. In this scheme, the algorithm partitions the training patterns to groups, where all patterns within each group share the same bandwidth. Grouping relies on the analysis of the local nearest neighbor distance information around the patterns and the principal component analysis with fuzzy clustering. Furthermore, we use a hybrid optimization procedure combining a very efficient variant of the particle swarm optimizer and a quasi-Newton method for global optimization and locally optimal fine-tuning of the network bandwidths. Training is based on the minimization of a flexible adaptation of the leave-one-out validation error that enhances the network generalization. We test the proposed algorithm with real and synthetic datasets, and results show that it exhibits competitive regression performance compared to other techniques.

In this paper, we propose a novel approach to stereo correspondence based on the optimization of a continuous disparity surface defined parametrically using radial basis functions. Principal advantages over other methods include the use of constrained nonlinear programming to perform regularization as a hierarchical multiobjective optimization which differs from the standard weighted sum approach, so that regularization becomes more consistent with the notion of Pareto optimality. Furthermore, the optimization algorithm is capable of handling arbitrary constraints on the sought parameters, so that a variety of types of a priori scene information can be incorporated explicitly to the problem definition. To exemplify this we derive a new continuous unary formulation of the disparity gradient limit constraint and propose other types of potential constraints for a priori knowledge. Furthermore, the optimization employs a smoothness oriented regularization operator to preserve surface discontinuities, a flexible block decomposition approach of the disparity surface to allow parallelization and a correlation-based fitting with heuristics to initialize the parameters and avoid local optima effectively. Experiments with standard stereo imagery show that the method handles adequately the imposed constraints and produces surfaces with accurate level of elevation detail.

This paper proposes a novel optimization algorithm for image-space matching and three-dimensional space analysis, using an adapted scheme of evolutionary computation that employs the concept of symbiosis in a collective of homogeneous populations. It is applied to the automatic generation of disparity surfaces used for depth estimation in stereo vision. The global task of approximating the complete disparity surface is decomposed to a large number of smaller local problems, each solvable by a smaller processing unit. Coevolution is sustained in such a way as to counteract the arbitrary decomposition of the original super-problem, so that the local evolutions of all the subproblems become interlocked. This, in the long run, provides a consistent global solution, and it does so via an asynchronous and massively parallel architecture. The entire surface is partitioned to a set of adjoining patches represented by distinct species or populations, with phenotypes corresponding to different polynomial functionals. The credit assignment functions take into account both self and symbiotic terms in an adaptive and dynamic manner, in order to produce disparity patches that are fit within their own domain and at the same time fit in association with their symbionts. This persistent propagation of local interactions to a global scale throughout evolution generates a unified disparity surface composed of the many smaller patch surfaces.