A commonly used distribution on the probability simplex is the Dirichlet distribution. In this paper we present the linear exponential family as an alternative. The distribution is known in the statistics community, but we present in this paper a numerically stable method to compute its parameters. Although the Dirichlet distribution is known to be a good Bayesian prior for probabilities we believe this paper shows that the linear exponential model offers a good alternative in other contexts, such as when we want to use posterior probabilities as features for automatic speech recognition. We show how to incorporate posterior probabilities as additional features to an existing GMM, and show that the resulting model gives a 0.6% relative gain on a broadcast news speech recognition system. Index Terms: Linear Exponential Family, Simplex, Divided Difference, Partition Function

We study the problem of learning a sparse linear regression vector under additional conditions on the structure of its sparsity pattern. We present a family of convex penalty functions, which encode this prior knowledge by means of a set of constraints on the absolute values of the regression coefficients. This family subsumes the $\ell_1$ norm and is flexible enough to include different models of sparsity patterns, which are of practical and theoretical importance. We establish some important properties of these functions and discuss some examples where they can be computed explicitly. Moreover, we present a convergent optimization algorithm for solving regularized least squares with these penalty functions. Numerical simulations highlight the benefit of structured sparsity and the advantage offered by our approach over the Lasso and other related methods.

Learning the common structure shared by a set of supervised tasks is an important practical and theoretical problem. Knowledge of this structure may lead to bet- ter generalization performance on the tasks and may also facilitate learning new tasks. We propose a framework for solving this problem, which is based on reg- ularization with spectral functions of matrices. This class of regularization prob- lems exhibits appealing computational properties and can be optimized ef(cid:2)ciently by an alternating minimization algorithm. In addition, we provide a necessary and suf(cid:2)cient condition for convexity of the regularizer. We analyze concrete ex- amples of the framework, which are equivalent to regularization with Lp matrix norms. Experiments on two real data sets indicate that the algorithm scales well with the number of tasks and improves on state of the art statistical performance.

We address the problem of learning a kernel for a given supervised learning task. Our approach consists in searching within the convex hull of a prescribed set of basic kernels for one which minimizes a convex regularization functional. A unique feature of this approach compared to others in the literature is that the number of basic kernels can be infinite. We only require that they are continuously parameterized. For example, the basic kernels could be isotropic Gaussians with variance in a prescribed interval or even Gaussians parameterized by multiple continuous parameters. Our work builds upon a formulation involving a minimax optimization problem and a recently proposed greedy algorithm for learning the kernel. Although this optimization problem is not convex, it belongs to the larger class of DC (difference of convex functions) programs. Therefore, we apply recent results from DC optimization theory to create a new algorithm for learning the kernel. Our experimental results on benchmark data sets show that this algorithm outperforms a previously proposed method.

In this paper we investigate conditions on the features of a continuous kernel so that it may approximate an arbitrary continuous target function uniformly on any compact subset of the input space. A number of concrete examples are given of kernels with this universal approximating property.

We study the problem of learning many related tasks simultaneously using kernel methods and regularization. The standard single-task kernel methods, such as support vector machines and regularization networks, are extended to the case of multi-task learning. Our analysis shows that the problem of estimating many task functions with regularization can be cast as a single task learning problem if a family of multi-task kernel functions we define is used. These kernels model relations among the tasks and are derived from a novel form of regularizers. Specific kernels that can be used for multi-task learning are provided and experimentally tested on two real data sets. In agreement with past empirical work on multi-task learning, the experiments show that learning multiple related tasks simultaneously using the proposed approach can significantly outperform standard single-task learning particularly when there are many related tasks but few data per task.

We study the problem of finding an optimal kernel from a prescribed convex set of kernels K for learning a real-valued function by regularization. We establish for a wide variety of regularization functionals that this leads to a convex optimization problem and, for square loss regularization, we characterize the solution of this problem. We show that, although K may be an uncountable set, the optimal kernel is always obtained as a convex combination of at most m+2 basic kernels, where m is the number of data examples. In particular, our results apply to learning the optimal radial kernel or the optimal dot product kernel.

In this letter, we provide a study of learning in a Hilbert space of vectorvalued functions. We motivate the need for extending learning theory of scalar-valued functions by practical considerations and establish some basic results for learning vector-valued functions that should prove useful in applications. Specifically, we allow an output space Y to be a Hilbert space, and we consider a reproducing kernel Hilbert space of functions whose values lie in Y. In this setting, we derive the form of the minimal norm interpolant to a finite set of data and apply it to study some regularization functionals that are important in learning theory. We consider specific examples of such functionals corresponding to multiple-output regularization networks and support vector machines, for both regression and classification. Finally, we provide classes of operator-valued kernels of the dot product and translation-invariant type.

This paper provides a foundation for multitask learning using reproducing ker- nel Hilbert spaces of vectorvalued functions. In this setting, the kernel is a matrixvalued function. Some explicit examples will be described which go be- yond our earlier results in [7]. In particular, we characterize classes of matrix valued kernels which are linear and are of the dot product or the translation invari- ant type. We discuss how these kernels can be used to model relations between the tasks and present linear multitask learning algorithms. Finally, we present a novel proof of the representer theorem for a minimizer of a regularization func- tional which is based on the notion of minimal norm interpolation.

We discuss several numerical methods for optimum feature selection for multivariate data based on maximum entropy and maximum likelihood criteria. Our point of view is to consider observed data x/sup 1/, x/sup 2/,..., x/sup N/ in R/sup d/ to be samples from some unknown pdf P. We project this data onto d directions, subsequently estimate the pdf of the univariate data, then find the maximum entropy (or likelihood) of all multivariate pdfs in R/sup d/ with marginals in these directions prescribed by the estimated univariate pdfs and finally maximize the entropy (or likelihood) further over the choice of these directions. This strategy for optimal feature selection depends on the method used to estimate univariate data.

We consider a parametric family of density functions of the type exp(-|x|/sup /spl alpha//2/) for modeling acoustic feature vectors used in automatic recognition of speech. The parameter /spl alpha/ is a measure of the impulsiveness as well as the nongaussian nature of the data. While previous work has focused on estimating the mean and the variance of the data here we attempt to estimate the impulsiveness /spl alpha/ from the data on a maximum likelihood basis. We show that there is a balance between /spl alpha/ and the number of data points N that must be satisfied before maximum likelihood estimation is carried out. Numerical experiments are performed on multidimensional vectors obtained from speech data.

Conjugate quadrature filters (CQF) with multiple zeros at 1 have classical applications to unitary subband coding of signals using exact reconstruction filter banks. Previous work shows how to construct, given a set of n negative numbers, a CQF whose degree does not exceed 2n-1 and whose zeros contain the specified negative numbers, and applies such filters to interpolatory subdivision and to wavelet construction in Sobelov spaces. This paper describes a previous result of the authors which extends this construction for an arbitrary set of n nonzero complex numbers that contains no negative or negative reciprocal conjugate pairs. Detailed derivations are given elsewhere. We design several filters using an exchange algorithm to illustrate a conjecture concerning the minimal degree and we discuss an application to coding transient acoustic signals.

class of search algorithms for locahng the root of a function is devised and studied.Each member of the class of algorithms is globally convergent as m the classical method of binary search However, the class contains members of higher order of convergence as well.These higher order methods require the existence of bounds on derivatives of the function whose root is sought and they proceed by exploiting these bounds.