Frank-Wolfe methods (FW) have gained significant interest in the machine learning community due to their ability to efficiently solve large problems that admit a sparse structure (e.g. sparse vectors and low-rank matrices). However the performance of the existing FW method hinges on the quality of the linear approximation. This typically restricts FW to smooth functions for which the approximation quality, indicated by a global curvature measure, is reasonably good. In this paper, we propose a modified FW algorithm amenable to nonsmooth functions, subject to a separability assumption, by optimizing for approximation quality over all affine functions, given a neighborhood of interest. We analyze theoretical properties of the proposed algorithm and demonstrate that it overcomes many issues associated with existing methods in the context of nonsmooth low-rank matrix estimation.

The Frank-Wolfe (FW) algorithm has been widely used in solving nuclear norm constrained problems, since it does not require projections. However, FW often yields high rank intermediate iterates, which can be very expensive in time and space costs for large problems. To address this issue, we propose a rank-drop method for nuclear norm constrained problems. The goal is to generate descent steps that lead to rank decreases, maintaining low-rank solutions throughout the algorithm. Moreover, the optimization problems are constrained to ensure that the rank-drop step is also feasible and can be readily incorporated into a projection-free minimization method, e.g., Frank-Wolfe. We demonstrate that by incorporating rank-drop steps into the Frank-Wolfe algorithm, the rank of the solution is greatly reduced compared to the original Frank-Wolfe or its common variants.

The Semi-Supervised Support Vector Machine (S3VM) solves a non-convex, Mixed-Integer Program (MIP). Due to difficulty in solving the problem, convex approximations have typically been used. However, existing approaches suffer from poor scalability and struggle on certain datasets, compared to graph based counterparts. The poor predictive performance suggests that for some datasets, convex approximations may not be a sufficiently accurate approximation to the problem. We present a self-training approach with self-adapting regularization parameters for S3VM formulations. At each iteration, the regularization parameters are adapted to better reflect label confidence, class proportion, and to gradually include more unlabeled points. We show that updating the S3VM framework iteratively in this fashion, the sequence of SVM subproblems can be solved very efficiently and the solution generated by this sequence yields superior performance compared to leading SSL methods.

Rare class problems are common in real-world applications across a wide range of domains. Standard classification algorithms are known to perform poorly in these cases, since they focus on overall classification accuracy. In addition, we have seen a significant increase of data in recent years, resulting in many large scale rare class problems. In this paper, we focus on nonlinear kernel based classification methods expressed as a regularized loss minimization problem. We address the challenges associated with both rare class problems and large scale learning, by 1) optimizing area under curve of the receiver of operator characteristic in the training process, instead of classification accuracy and 2) using a rare class kernel representation to achieve an efficient time and space algorithm. We call the algorithm RankRC. We provide justifications for the rare class representation and experimentally illustrate the effectiveness of RankRC in test performance, computational complexity, and model robustness.

We consider an infinite mixture model of Gaussian processes that share mixture components between non-local clusters in data. Meeds and Osindero (2006) use a single Dirichlet process prior to specify a mixture of Gaussian processes using an infinite number of experts. In this paper, we extend this approach to allow for experts to be shared non-locally across the input domain. This is accomplished with a hierarchical double Dirichlet process prior, which builds upon a standard hierarchical Dirichlet process by incorporating local parameters that are unique to each cluster while sharing mixture components between them. We evaluate the model on simulated and real data, showing that sharing Gaussian process components non-locally can yield effective and useful models for richly clustered non-stationary, non-linear data.

We consider the problem of hedging the loss of a given portfolio of derivatives using a set of more liquid derivative instruments. We illustrate why the typical mathematical formulation for this hedging problem is ill-posed. We propose to determine a hedging portfolio by minimizing a proportional cost subject to an upper bound on the hedge risk; this bound is typically slightly larger than the optimal hedge risk achievable without cost consideration. We illustrate that the optimal hedging portfolio obtained by the proposed method is attractive since it consists of fewer instruments with a comparable risk. Finally we illustrate the importance of modeling volatility uncertainty in hedge risk minimization.

A reliable and efficient computational algorithm for restoring blurred and noisy images is proposed. The restoration process is based on the minimal total variation principle introduced by Rudin et al. For discrete images, the proposed algorithm minimizes a piecewise linear l (1) function (a measure of total variation) subject to a single 2-norm inequality constraint (a measure of data fit). The algorithm starts by finding a feasible point for the inequality constraint using a (partial) conjugate gradient method. This corresponds to a deblurring process. Noise and other artifacts are removed by a subsequent total variation minimization process. The use of the linear l(1) objective function for the total variation measurement leads to a simpler computational algorithm. Both the steepest descent and an affine scaling Newton method are considered to solve this constrained piecewise linear l(1) minimization problem. The resulting algorithm, when viewed as an image restoration and enhancement process, has the feature that it can be used in an adaptive/interactive manner in situations when knowledge of the noise variance is either unavailable or unreliable. Numerical examples are presented to demonstrate the effectiveness of the proposed iterative image restoration and enhancement process.