BACKGROUND: Hinge-bending movements in proteins comprising two or more domains form a large class of functional movements. Hinge-bending regions demarcate protein domains and collectively control the domain movement. Consequently, the ability to recognise sequence features of hinge-bending regions and to be able to predict them from sequence alone would benefit various areas of protein research. For example, an understanding of how the sequence features of these regions relate to dynamic properties in multi-domain proteins would aid in the rational design of linkers in therapeutic fusion proteins. RESULTS: The DynDom database of protein domain movements comprises sequences annotated to indicate whether the amino acid residue is located within a hinge-bending region or within an intradomain region. Using statistical methods and Kernel Logistic Regression (KLR) models, this data was used to determine sequence features that favour or disfavour hinge-bending regions. This is a difficult classification problem as the number of negative cases (intradomain residues) is much larger than the number of positive cases (hinge residues). The statistical methods and the KLR models both show that cysteine has the lowest propensity for hinge-bending regions and proline has the highest, even though it is the most rigid amino acid. As hinge-bending regions have been previously shown to occur frequently at the terminal regions of the secondary structures, the propensity for proline at these regions is likely due to its tendency to break secondary structures. The KLR models also indicate that isoleucine may act as a domain-capping residue. We have found that a quadratic KLR model outperforms a linear KLR model and that improvement in performance occurs up to very long window lengths (eighty residues) indicating long-range correlations. CONCLUSION: In contrast to the only other approach that focused solely on interdomain hinge-bending regions, the method provides a modest and statistically significant improvement over a random classifier. An explanation of the KLR results is that in the prediction of hinge-bending regions a long-range correlation is at play between a small number amino acids that either favour or disfavour hinge-bending regions. The resulting sequence-based prediction tool, HingeSeek, is available to run through a webserver at hingeseek.cmp.uea.ac.uk.

Tuning the regularisation and kernel hyperparameters is a vital step in optimising the generalisation performance of kernel methods, such as the support vector machine (SVM). This is most often performed by minimising a resampling/cross-validation based model selection criterion, however there seems little practical guidance on the most suitable form of resampling. This paper presents the results of an extensive empirical evaluation of resampling procedures for SVM hyperparameter selection, designed to address this gap in the machine learning literature. We tested 15 different resampling procedures on 121 binary classification data sets in order to select the best SVM hyperparameters. We used three very different statistical procedures to analyse the results: the standard multi- classifier/multi-data set procedure proposed by Dem\v{s}ar, the confidence intervals on the excess loss of each procedure in relation to 5-fold cross validation, and the Bayes factor analysis proposed by Barber. We conclude that a 2-fold procedure is appropriate to select the hyperparameters of an SVM for data sets for 1000 or more datapoints, while a 3-fold procedure is appropriate for smaller data sets.

ChaLearn is organizing the Automatic Machine Learning (AutoML) contest for IJCNN 2015, which challenges participants to solve classification and regression problems without any human intervention. Participants' code is automatically run on the contest servers to train and test learning machines. However, there is no obligation to submit code; half of the prizes can be won by submitting prediction results only. Datasets of progressively increasing difficulty are introduced throughout the six rounds of the challenge. (Participants can enter the competition in any round.) The rounds alternate phases in which learners are tested on datasets participants have not seen, and phases in which participants have limited time to tweak their algorithms on those datasets to improve performance. This challenge will push the state of the art in fully automatic machine learning on a wide range of real-world problems. The platform will remain available beyond the termination of the challenge.

Semi-supervised learning involves constructing predictive models with both labelled and unlabelled training data. The need for semi-supervised learning is driven by the fact that unlabelled data are often easy and cheap to obtain, whereas labelling data requires costly and time consuming human intervention and expertise. Semi-supervised methods commonly use self training, which involves using the labelled data to predict the unlabelled data, then iteratively reconstructing classifiers using the predicted labels. Our aim is to determine whether self training classifiers actually improves performance. Expectation maximization is a commonly used self training scheme. We investigate whether an expectation maximization scheme improves a naïve Bayes classifier through experimentation with 36 discrete and 20 continuous real world benchmark UCI datasets. Rather surprisingly, we find that in practice the self training actually makes the classifier worse. The cause for this detrimental affect on performance could either be with the self training scheme itself, or how self training works in conjunction with the classifier. Our hypothesis is that it is the latter cause, and the violation of the naïve Bayes model assumption of independence of attributes means predictive errors propagate through the self training scheme. To test whether this is the case, we generate simulated data with the same attribute distribution as the UCI data, but where the attributes are independent. Experiments with this data demonstrate that semi-supervised learning does improve performance, leading to significantly more accurate classifiers. These results demonstrate that semi-supervised learning cannot be applied blindly without considering the nature of the classifier, because the assumptions implicit in the classifier may result in a degradation in performance.

MOTIVATION: A popular method for classification of protein domain movements apportions them into two main types: those with a 'hinge' mechanism and those with a 'shear' mechanism. The intuitive assignment of domain movements to these classes has limited the number of domain movements that can be classified in this way. Furthermore, whether intended or not, the term 'shear' is often interpreted to mean a relative translation of the domains. RESULTS: Numbers of occurrences of four different types of residue contact changes between domains were optimally combined by logistic regression using the training set of domain movements intuitively classified as hinge and shear to produce a predictor for hinge and shear. This predictor was applied to give a 10-fold increase in the number of examples over the number previously available with a high degree of precision. It is shown that overall a relative translation of domains is rare, and that there is no difference between hinge and shear mechanisms in this respect. However, the shear set contains significantly more examples of domains having a relative twisting movement than the hinge set. The angle of rotation is also shown to be a good discriminator between the two mechanisms. AVAILABILITY AND IMPLEMENTATION: Results are free to browse at http://www.cmp.uea.ac.uk/dyndom/interface/. CONTACT: [email protected]. SUPPLEMENTARY INFORMATION: Supplementary data are available at Bioinformatics online.

This paper presents some initial results of an investigation into the use of machine learning methods to detect natively disordered regions in proteins from sequence information. A committee of Relevance Vector Machines is used to select the optimal window size for residue-by-residue prediction of disordered regions, based on local amino-acid composition. The minimal error rate of ≈ 15% is achieved using very long (205 residue) window lengths, with the classifier making little use of more local sequence information. This suggests that disorder arises principally due to large scale diffuse changes in mean hydropathy and to a lesser extent mean charge. We also demonstrate that the proportion of proteins having long disordered regions in operational conditions cannot be reliably estimated using a classifier trained on a balanced dataset.

We organized a data mining challenge on “active learning” for IJCNN/WCCI 2010, addressing machine learning problems where labeling data is expensive, but large amounts of unlabeled data are available at low cost. Examples include handwriting and speech recognition, document classification, vision tasks, drug design using recombinant molecules and protein engineering. Such problems might be tackled from different angles: learning from unlabeled data or active learning. In the former case, the algorithms must satisfy themselves with the limited amount of labeled data and capitalize on the unlabeled data with semi-supervised learning methods. Several challenges have addressed this problem in the past. In the latter case, the algorithms may place a limited number of queries to get new sample labels. The goal in that case is to optimize the queries and the problem is referred to as active learning. While the problem of active learning is of great importance, organizing a challenge in that area is non trivial. This is the problem we have addressed, and we describe our approach in this paper. The “active learning” challenge is part of the WCCI 2010 competition program (http://www.wcci2010. org/competition-program). The website of the challenge remains open for submission of new methods beyond the termination of the challenge as a resource for students and researchers (http://clopinet.com/al).

The generalised linear model (GLM) is the standard approach in classical statistics for regression tasks where it is appropriate to measure the data misfit using a likelihood drawn from the exponential family of distributions. In this paper, we apply the kernel trick to give a non-linear variant of the GLM, the generalised kernel machine (GKM), in which a regularised GLM is constructed in a fixed feature space implicitly defined by a Mercer kernel. The MATLAB symbolic maths toolbox is used to automatically create a suite of generalised kernel machines, including methods for automated model selection based on approximate leave-one-out cross-validation. In doing so, we provide a common framework encompassing a wide range of existing and novel kernel learning methods, and highlight their connections with earlier techniques from classical statistics. Examples including kernel ridge regression, kernel logistic regression and kernel Poisson regression are given to demonstrate the flexibility and utility of the generalised kernel machine.

The optimal model parameters of a kernel machine are typically given by the solution of a convex optimisation problem with a single global optimum. Obtaining the best possible performance is therefore largely a matter of the design of a good kernel for the problem at hand, exploiting any underlying structure and optimisation of the regularisation and kernel parameters, i.e. model selection. Fortunately, analytic bounds on, or approximations to, the leave-one-out cross-validation error are often available, providing an efficient and generally reliable means to guide model selection. However, the degree to which the incorporation of prior knowledge improves performance over that which can be obtained using "standard" kernels with automated model selection (i.e. agnostic learning), is an open question. In this paper, we compare approaches using example solutions for all of the benchmark tasks on both tracks of the IJCNN-2007 Agnostic Learning versus Prior Knowledge Challenge.

"When everything fails, ask for additional domain knowledge" is the current motto of machine learning. Therefore, assessing the real added value of prior/domain knowledge is a both deep and practical question. Most commercial data mining programs accept data pre-formatted as a table, each example being encoded as a fixed set of features. Is it worth spending time engineering elaborate features incorporating domain knowledge and/or designing ad hoc algorithms? Or else, can off-the-shelf programs working on simple features encoding the raw data without much domain knowledge do as well or better than skilled data analysts? To answer these questions, we organized a challenge for IJCNN 2007. The participants were allowed to compete in two tracks: The "prior knowledge" (PK) track, for which they had access to the original raw data representation and as much knowledge as possible about the data, and the "agnostic learning" (AL) track for which they were forced to use data pre-formatted as a table with dummy features. The AL vs. PK challenge Web site remains open: http://www.agnostic.inf.ethz.ch/.

J.-H. Chen and C.-S. Chen have recently proposed a nonlinear variant of Keller and Hunt's fuzzy perceptron algorithm, based on the now familiar "kernel trick." In this letter, we demonstrate experimentally that J.-H. Chen and C.-S. Chen's assertion that the fuzzy kernel perceptron (FKP) outperforms the support vector machine (SVM) cannot be sustained. A more thorough model comparison exercise, based on a much wider range of benchmark data sets, shows that the FKP algorithm is not competitive with the SVM.

While the model parameters of many kernel learning methods are given by the solution of a convex optimisation problem, the selection of good values for the kernel and regularisation parameters, i.e. model selection, is much less straight-forward. This paper describes a simple and efficient approach to model selection for weighted least-squares support vector machines, and compares a variety of model selection criteria based on leave-one-out cross-validation. An external cross-validation procedure is used for performance estimation, with model selection performed independently in each fold to avoid selection bias. The best entry based on these methods was ranked in joint first place in the WCCI-2006 performance prediction challenge, demonstrating the effectiveness of this approach.

Artificial neural networks have proved an attractive approach to non-linear regression problems arising in environmental modelling, such as statistical downscaling, short-term forecasting of atmospheric pollutant concentrations and rainfall run-off modelling. However, environmental datasets are frequently very noisy and characterised by a noise process that may be heteroscedastic (having input dependent variance) and/or non-Gaussian. The aim of this paper is to review an existing methodology for estimating predictive uncertainty in such situations, and more importantly illustrate how a model of the predictive distribution may be exploited in assessing the possible impacts of climate change and to improve current decision making processes. The results of the WCCI-2006 predictive uncertainty in environmental modelling challenge are also reviewed and some areas suggested where further research may provide significant benefits.

Multinomial logistic regression provides the standard penalised maximum- likelihood solution to multi-class pattern recognition problems. More recently, the development of sparse multinomial logistic regression models has found ap- plication in text processing and microarray classification, where explicit identifi- cation of the most informative features is of value. In this paper, we propose a sparse multinomial logistic regression method, in which the sparsity arises from the use of a Laplace prior, but where the usual regularisation parameter is inte- grated out analytically. Evaluation over a range of benchmark datasets reveals this approach results in similar generalisation performance to that obtained using cross-validation, but at greatly reduced computational expense.

MOTIVATION: Gene selection algorithms for cancer classification, based on the expression of a small number of biomarker genes, have been the subject of considerable research in recent years. Shevade and Keerthi propose a gene selection algorithm based on sparse logistic regression (SLogReg) incorporating a Laplace prior to promote sparsity in the model parameters, and provide a simple but efficient training procedure. The degree of sparsity obtained is determined by the value of a regularization parameter, which must be carefully tuned in order to optimize performance. This normally involves a model selection stage, based on a computationally intensive search for the minimizer of the cross-validation error. In this paper, we demonstrate that a simple Bayesian approach can be taken to eliminate this regularization parameter entirely, by integrating it out analytically using an uninformative Jeffrey's prior. The improved algorithm (BLogReg) is then typically two or three orders of magnitude faster than the original algorithm, as there is no longer a need for a model selection step. The BLogReg algorithm is also free from selection bias in performance estimation, a common pitfall in the application of machine learning algorithms in cancer classification. RESULTS: The SLogReg, BLogReg and Relevance Vector Machine (RVM) gene selection algorithms are evaluated over the well-studied colon cancer and leukaemia benchmark datasets. The leave-one-out estimates of the probability of test error and cross-entropy of the BLogReg and SLogReg algorithms are very similar, however the BlogReg algorithm is found to be considerably faster than the original SLogReg algorithm. Using nested cross-validation to avoid selection bias, performance estimation for SLogReg on the leukaemia dataset takes almost 48 h, whereas the corresponding result for BLogReg is obtained in only 1 min 24 s, making BLogReg by far the more practical algorithm. BLogReg also demonstrates better estimates of conditional probability than the RVM, which are of great importance in medical applications, with similar computational expense. AVAILABILITY: A MATLAB implementation of the sparse logistic regression algorithm with Bayesian regularization (BLogReg) is available from http://theoval.cmp.uea.ac.uk/~gcc/cbl/blogreg/

Survival analysis is a branch of statistics concerned with the time elapsing before "failure," with diverse applications in medical statistics and the analysis of the reliability of electrical or mechanical components. We introduce a parametric accelerated life survival analysis model based on kernel learning methods that, at least in principal, is able to learn arbitrary dependencies between a vector of explanatory variables and the scale of the distribution of survival times. The proposed kernel survival analysis method is then used to model the growth domain of Clostridium botulinum, the food processing and storage conditions permitting the growth of this foodborne microbial pathogen, leading to the production of the neurotoxin responsible for botulism. A Bayesian training procedure, based on the evidence framework, is used for model selection and to provide a credible interval on model predictions. The kernel survival analysis models are found to be more accurate than models based on more traditional survival analysis techniques but also suggest a risk assessment of the foodborne botulism hazard would benefit from the collection of additional data.

We introduce a simple framework for sparse Bayesian learning with multi-layer perceptron (IMLP) networks, inspired by Tipping's relevance vector machine (RVM). Like the RVM, a Bayesian prior is adopted that includes separate hyperparameters for each weight, allowing redundant weights and hidden layer units to be identified and subsequently pruned from the network, whilst also providing a means to avoid over-fitting the training data. This approach is also more easily implemented, as only the diagonal elements of the Hessian matrix are used in the update formula for the regularisation parameters, rather than the traces of square sub-matrices of the Hessian corresponding to the weights associated with each regularisation parameter. The proposed relevance multi-layer perceptron (RMLP) is evaluated over several publicly available benchmark datasets, demonstrating the viability of the approach, giving rise to similar generalisation performance, but with far fewer weights.

In this paper, we present a simple mathematical trick that simplifies the derivation of Bayesian treatments of a variety of sparse kernel learning methods. The incomplete Cholesky factorisation due to (Fine and Scheinberg, 2001) is used to transform the dual parameter space, such that the covariance matrix of the Gaussian prior over model parameters becomes the identity matrix. The regularisation term is then the familiar weight-decay regulariser, allowing the Bayesian analysis to proceed straight-forwardly via the methods developed by MacKay (1992). As a bye-product, the incomplete Cholesky factorisation algorithm also identifies a subset of the training data forming an approximate basis for the remaining data in feature space, resulting in a sparse model. Bayesian treatments of the kernel ridge regression algorithm (Saunders et al., 1998), with both constant and input dependent variance structures, arc given as illustrative examples of the proposed technique, which we hope will be more widely applicable.

The ability to predict the transcriptional regulation of genes, based on the composition of the upstream promoter region, would be a useful step in deciphering gene regulatory networks in eukaryotic organisms. In this paper we perform optimally regularised kernel Fisher discriminant (ORKFD) analysis of the upstream promoter sequences of genes to predict whether they are up- or down-regulated in response to glucose in the model plant Arahuiopsis thaliana. Three feature selection strategies are investigated, namely use of known promoter motifs drawn from the PLACE database, explicit enumeration of all possible k-mers and the use of the mismatch kernels (which effectively permits the construction of a linear model in the space of all possible k-mers with up to in mismatches). The leave-one-out cross-validation (LOOCV) error rate indicates that approximately two-thirds of the observed regulatory behaviour can be inferred by the presence of particular motifs in the upstream promoter sequence. The analysis has yielded novel biological insight, which has since been confirmed experimentally in vivo.

Two dimensional (2D) shape and appearance models are applied to the problem of creating a near-videorealistic talking head. A speech corpus of a talker uttering a set of phonetically balanced training sentences is analysed using a generative model of the human face. Segments of original parameter trajectories, corresponding to the synthesis unit (e.g.~triphone), are extracted from a codebook, then normalised, blended, concatenated and smoothed before being applied to the model to give natural, realistic animations of novel utterances. The system provides a 2D image sequence corresponding to the face of a talker. It is also used to animate the face of a 3D avatar by displacing the mesh according to movements of points in the shape model and dynamically texturing the face polygons using the appearance model.

We present work towards videorealistic synthetic visual speech using non-rigid appearance models. These models are used to track a talking face enunciating a set of training sentences. The resultant parameter trajectories are used in a concatenative synthesis scheme, where samples of original data are extracted from a corpus and concatenated to form new unseen sequences. Here we explore the effect on the synthesiser output of blending several synthesis units considered similar to the desired unit. We present preliminary subjective and objective results used to judge the realism of the system.

This paper compares the performance, in terms of prediction accuracy, of a learning classifier system based on Wilson's XCS with commonly used classifiers from the fields of decision trees, neural networks and support vector machines. The experiments are performed on the Forest Cover Type database, a large data set available at the UCI KDD Archive. The first objective of this paper is to highlight the potential of XCS as a data mining tool. The second objective is to provide extensive benchmarking results for experiments performed under randomised conditions for several modelling techniques. We find that C5 Decision trees perform significantly better than other techniques, and that the learning classifier system performs better or as well as three of the eight classifiers used. We discuss why C5 outperforms the other classifiers and identify ways in which XCS could be adapted to make it more suitable for data mining.

Geophysical surveys form a vital component of modern archaeological studies, providing data on the location of surviving sub-surface remains. Unfortunately interpretation of the raw output from geophysical surveys requires considerable expertise and current methods for machine interpretation are computationally expensive. In this paper, we describe the use of multi-layer perceptron neural networks from automated interpretation of geophysical survey data and present promising initial results for a survey taken of the Roman suburb of Butrint, a classical city situated in what is now Albania. Clearly a computationally efficient and straightforward means of inverting geophysical survey data is potentially an extremely useful tool in encouraging public awareness of cultural heritage.

Clostridium botulinum is a bacterium present in the raw ingredients of many foods. It produces a powerful neurotoxin as part of its growth process, that can prove fatal when doses as small as 30ng are consumed. It is therefore vital to be able to accurately determine the food processing and storage conditions where toxin production is possible, known as the "growth domain". This paper describes a new approach to modelling the growth domain of microbial pathogens, by constructing a regularised kernel model relating heat treatment and subsequent incubation conditions to the parameters of a statistical distribution modelling the probability of growth as a function of incubation time. We demonstrate that the use of the "kernel trick" permits the extension of methods from classical survival analysis to account for non-linear dependencies in a principled manner.

In previous work, we experimented with different techniques of vector-based call routing, using the transcriptions of the queries to compare algorithms. In this paper, we base the routing decisions on the recogniser output rather than transcriptions and examine the use of confidence measures (CMs) to combat the problems caused by the "noise" in the recogniser output. CMs are derived for both the words output from the recogniser and for the routings themselves and are used to investigate improving both routing accuracy and routing confidence. Results are given for a 35 route retail store enquiry-point task. They suggest that although routing error is controlled by the recogniser error-rate, confidence in routing decisions can be improved using these techniques.

In this paper we present initial work towards a video-realistic visual speech synthesiser based on statistical models of shape and appearance. A synthesised image sequence corresponding to an utterance is formed by concatenation of synthesis units (in this case phonemes) from a pre-recorded corpus of training data. A smoothing spline is applied to the concatenated parameters to ensure smooth transitions between frames and the resultant parameters applied to the model—early results look promising.

An analysis of predictability of a nonlinear and nonstationary ozone time series is provided. For rigour, the deterministic versus stochastic (DVS) analysis is first undertaken to detect and measure inherent nonlinearity of the data. Based upon this, neural and linear adaptive predictors are compared on this time series for various filter orders, hence indicating the embedding dimension. Simulation results confirm the analysis and show that for this class of air pollution data, neural, especially recurrent neural predictors, perform best.

Recently Schraudolph (1999) described an ingenious, fast, and compact approximation of the exponential function through manipulation of the components of a standard (IEEE-754 (IEEE, 1985)) floating-point representation. This brief note communicates a recoding of this procedure that overcomes some of the limitations of the original macro at little or no additional computational expense.

A useful representation of an image would be an object tree in which nodes represent objects, or parts of objects, and which includes at least one node that, together with its children, represents each object: a grandmother node. It is shown that scale-trees, obtained from greyscale images, approximate such a tree. It is then shown how they may be modified using other attributes to more closely become object trees. The result is a data structure that pro-vides “handles ” for every element of the image that can be used for manipu-lating the image. This segmentation has potential for object recognition. 1

Vector quantisation is a widely used technique in low-bit rate coding of speech and image data, but is highly sensitive to noise in the transmission channel. If the reference vector recalled by a corrupted index differs greatly from the intended reference vector, image quality can be degraded quite dramatically. The index assignment (IA) process attempts to re-order the code book so as to minimise the effects of errors introduced in the transmission channel, by assigning indices with similar binary patterns to similar reference vectors, usually at considerable computational expense. This paper describes a fast, novel index assignment algorithm based on Hall's solution (1970) to the quadratic assignment problem.

Accurate modeling of co-articulation, the contextsensitive merging of the boundaries between allophones in continuous speech, is vital for natural sounding speech synthesis. This paper describes initial research investigating the use of Bezier Curves to form models of co-articulation in human speech. A 12th order, pitch synchronous line spectral pair (LSP) [1] analysis is performed on a corpus of 239 phonetically balanced sentences of English speech. The resulting data are divided to form an inventory of the diphones occurring in the speech database. The trajectory of each line spectral pair parameter through each diphone can then be represented by a single cubic Bezier curve segment, found using the LevenbergMarquardt curve fitting method [2, 3]. Results are presented showing the accuracy of Bezier models of the coarticulation between different types of speech sounds.