When the data are high dimensional, widely used multivariate statistical methods such as principal component analysis can behave in unexpected ways. In settings where the dimension of the observations is comparable to the sample size, upward bias in sample eigenvalues and inconsistency of sample eigenvectors are among the most notable phenomena that appear. These phenomena, and the limiting behavior of the rescaled extreme sample eigenvalues, have recently been investigated in detail under the spiked covariance model. The behavior of the bulk of the sample eigenvalues under weak distributional assumptions on the observations has been described. These results have been exploited to develop new estimation and hypothesis testing methods for the population covariance matrix. Furthermore, partly in response to these phenomena, alternative classes of estimation procedures have been developed by exploiting sparsity of the eigenvectors or the covariance matrix. This paper gives an orientation to these areas.

Many papers in the hyperspectral literature use simulations (based on a linear mixture model) to test algorithms which either estimate the dimensionality of the data or endmem-bers. Typically these simulations use (i) “real world” end-members, (ii) proportions distributed according to a uniform or Dirichlet distribution on the endmember simplex, and (iii) Gaussian errors which are “spectrally” and “spatially” uncor-related. When the error standard deviations (SDs) in different bands are assumed to be unequal, they are usually estimated using Roger's method. The simulated and real world data in these papers are so different that one can't be confident that the various advocated methods work well with real world data. We propose a methodology which produces more realistic simulations, providing us with greater insights into the strengths and weaknesses of various advocated methods. In particular, using an AVIRIS Cuprite scene, we demonstrate that Roger's SD estimates are positively biased.

Compressed sensing posits that, within limits, one can undersample a sparse signal and yet reconstruct it accurately. Knowing the precise limits to such undersampling is important both for theory and practice. We present a formula that characterizes the allowed undersampling of generalized sparse objects. The formula applies to approximate message passing (AMP) algorithms for compressed sensing, which are here generalized to employ denoising operators besides the traditional scalar soft thresholding denoiser. This paper gives several examples including scalar denoisers not derived from convex penalization-the firm shrinkage nonlinearity and the minimax nonlinearity-and also nonscalar denoisers-block thresholding, monotone regression, and total variation minimization. Let the variables ε = k/N and δ = n/N denote the generalized sparsity and undersampling fractions for sampling the k-generalized-sparse N-vector x0according to y=Ax0. Here, A is an n×N measurement matrix whose entries are iid standard Gaussian. The formula states that the phase transition curve δ = δ(ε) separating successful from unsuccessful reconstruction of x0by AMP is given by δ = M(ε|Denoiser) where M(ε|Denoiser) denotes the per-coordinate minimax mean squared error (MSE) of the specified, optimally tuned denoiser in the directly observed problem y = x + z. In short, the phase transition of a noiseless undersampling problem is identical to the minimax MSE in a denoising problem. We prove that this formula follows from state evolution and present numerical results validating it in a wide range of settings. The above formula generates numerous new insights, both in the scalar and in the nonscalar cases.

We consider the compressed sensing problem where the object x0∈ ℝNis to be recovered from incomplete measurements y = Ax0+z. Here the sensing matrix A is an n×N random matrix with Gaussian entries and n1-penalized least-squares reconstruction (aka LASSO, Basis Pursuit). Suppose that xοis sparse in the sense of having ℓρnorm bounded by ξ · N1/ρfor some fixed 0; 0. In both the noisy (ziiid N(0, σ2)) and noiseless (z = 0) cases, we evaluate the worst-case asymptotic mean square error (AMSE) for optimally tuned ℓ1penalized least-squares, and we exhibit the least-favorable object xο(hardest sparse signal to recover) and the maximin penalization. Our explicit formulas yield precise relations. For example, in the noiseless case z = 0, for vectors xοof ℓρnorm bounded by 1, we show that mm max∥ x̑λ- x0∥2= n1-2/p(2log(N/n))2/p-1{1+oN(l)} where n, N → ∞, n/N → 0 slowly. The complete formulas applies to a general scaling limit n/N → δ and sparsity parameter ξ, and unexpectedly involve quantities from statistical decision theory. This reflects a deep connection between ℓ1-penalized ℓ2minimization and scalar soft thresholding.