The advent of large-scale inference has spurred reexamination of conventional statistical thinking. In a series of highly original articles, Efron persuasively illustrated the danger for downstream inference in assuming the veracity of a posited null distribution. In a Gaussian model for n many z-scores with at most$k \lt \frac {n}{2}$nonnulls, Efron suggests estimating the parameters of an empirical null$N(\theta , \sigma ^{2})$instead of assuming the theoretical null$N(0, 1)$. Looking to the robust statistics literature by viewing the nonnulls as outliers is unsatisfactory as the question of optimal rates is still open; even consistency is not known in the regime$k \asymp n$which is especially relevant to many large-scale inference applications. However, provably rate-optimal robust estimators have been developed in other models (e.g. Huber contamination) which appear quite close to Efron’s proposal. Notably, the impossibility of consistency when$k \asymp n$in these other models may suggest the same major weakness afflicts Efron’s popularly adopted recommendation. A sound evaluation thus requires a complete understanding of information-theoretic limits. We characterize the regime of k for which consistent estimation is possible, notably without imposing any assumptions at all on the nonnull effects. Unlike in other robust models, it is shown consistent estimation of the location parameter is possible if and only if$\frac {n}{2} {-} k = \omega (\sqrt {n})$, and of the scale parameter in the entire regime$k \lt \frac {n}{2}$. Furthermore, we establish sharp minimax rates and show estimators based on the empirical characteristic function are optimal by exploiting the Gaussian character of the data.

Sparse additive models are an attractive choice in circumstances calling for modelling flexibility in the face of high dimensionality. We study the signal detection problem and establish the minimax separation rate for the detection of a sparse additive signal. Our result is nonasymptotic and applicable to the general case where the univariate component functions belong to a generic reproducing kernel Hilbert space. Unlike the estimation theory, the minimax separation rate reveals a nontrivial interaction between sparsity and the choice of function space. We also investigate adaptation to sparsity and establish an adaptive testing rate for a generic function space; adaptation is possible in some spaces while others impose an unavoidable cost. Finally, adaptation to both sparsity and smoothness is studied in the setting of Sobolev space, and we correct some existing claims in the literature.

We study the phase synchronization problem with noisy measurements$Y=z^{*}z^{* { \mathrm {\scriptscriptstyle H} }}+\sigma W\in \mathbb {C}^{n\times n}$, where$z^{*}$is an$n$-dimensional complex unit-modulus vector and$W$is a complex-valued Gaussian random matrix. It is assumed that each entry$Y_{jk}$is observed with probability$p$. We prove that an SDP relaxation of the MLE achieves the error bound$(1+o(1))\frac {\sigma ^{2}}{2np}$under a normalized squared$\ell _{2}$loss. This result matches the minimax lower bound of the problem, and even the leading constant is sharp. The analysis of the SDP is based on an equivalent non-convex programming whose solution can be characterized as a fixed point of the generalized power iteration lifted to a higher dimensional space. This viewpoint unifies the proofs of the statistical optimality of three different methods: MLE, SDP, and generalized power method. The technique is also applied to the analysis of the SDP for$\mathbb {Z}_{2}$synchronization, and we achieve the minimax optimal error$\exp \left ({-(1-o(1))\frac {np}{2\sigma ^{2}}}\right)$with a sharp constant in the exponent.

We study the phase synchronization problem with measurements${Y}= {z}^{\ast} {z}^{\ast{\mathrm {H}}}+\sigma {W}\in \mathbb {C}^{n}\times {n}$, where${z}^{\ast}$is an${n}$-dimensional complex unit-modulus vector and${W}$is a complex-valued Gaussian random matrix. It is assumed that each entry${Y}_{jk}$is observed with probability${p}$. We prove that the minimax lower bound of estimating${z}^{\ast}$under the squared$\ell _{2}$loss is$(1- {o}(1))\frac {\sigma ^{2}}{2p}$. We also show that both generalized power method and maximum likelihood estimator achieve the error bound$(1+ {o}(1))\frac {\sigma ^{2}}{2p}$. Thus,$\frac {\sigma ^{2}}{2p}$is the exact asymptotic minimax error of the problem. Our upper bound analysis involves a precise characterization of the statistical property of the power iteration. The lower bound is derived through an application of van Trees’ inequality.