We study active learning methods for single index models of the form $F({\bm x}) = f(⟨{\bm w}, {\bm x}⟩)$, where $f:\mathbb{R} \to \mathbb{R}$ and ${\bx,\bm w} \in \mathbb{R}^d$. In addition to their theoretical interest as simple examples of non-linear neural networks, single index models have received significant recent attention due to applications in scientific machine learning like surrogate modeling for partial differential equations (PDEs). Such applications require sample-efficient active learning methods that are robust to adversarial noise. I.e., that work even in the challenging agnostic learning setting. We provide two main results on agnostic active learning of single index models. First, when $f$ is known and Lipschitz, we show that $\tilde{O}(d)$ samples collected via {statistical leverage score sampling} are sufficient to learn a near-optimal single index model. Leverage score sampling is simple to implement, efficient, and already widely used for actively learning linear models. Our result requires no assumptions on the data distribution, is optimal up to log factors, and improves quadratically on a recent ${O}(d^{2})$ bound of Gajjar et. al 2023. Second, we show that $\tilde{O}(d)$ samples suffice even in the more difficult setting when $f$ is \emph{unknown}. Our results leverage tools from high dimensional probability, including Dudley’s inequality and dual Sudakov minoration, as well as a novel, distribution-aware discretization of the class of Lipschitz functions.

In a great number of tasks in science and engineering, the goal is to infer an unknown image from a small number of noisy measurements collected from a known forward model describing certain sensing or imaging modality. Due to resource constraints, this image reconstruction task is often extremely ill-posed, which necessitates the adoption of expressive prior information to regularize the solution space. Score-based diffusion models, thanks to its impressive empirical success, have emerged as an appealing candidate of an expressive prior in image reconstruction. In order to accommodate diverse tasks at once, it is of great interest to develop efficient, consistent and robust algorithms that incorporate unconditional score functions of an image prior distribution in conjunction with flexible choices of forward models. This work develops an algorithmic framework for employing score-based diffusion models as an expressive data prior in nonlinear inverse problems with general forward models. Motivated by the plug-and-play framework in the imaging community, we introduce a diffusion plug-and-play method (DPnP) that alternatively calls two samplers, a proximal consistency sampler based solely on the likelihood function of the forward model, and a denoising diffusion sampler based solely on the score functions of the image prior. The key insight is that denoising under white Gaussian noise can be solved rigorously via both stochastic (i.e., DDPM-type) and deterministic (i.e., DDIM-type) samplers using the same set of score functions trained for generation. We establish both asymptotic and non-asymptotic performance guarantees of DPnP, and provide numerical experiments to illustrate its promise in solving both linear and nonlinear image reconstruction tasks. To the best of our knowledge, DPnP is the first provably-robust posterior sampling method for nonlinear inverse problems using unconditional diffusion priors.

We study the problem of reconstructing a continuous multidimensional signal from a small number of samples under Fourier constraints assuming that the Fourier power spectrum of the signal has some desirable properties, e.g. being compactly supported, being sparse. We further assume that the Fourier constraint can be expressed as a prior distribution on the Fourier power spectrum, which subsumes the aforementioned examples. The study of sampling and reconstructing in this vein has attracted much attention with a long history. In this paper, we are interested in finding oblivious sampling strategies, that is, sampling without knowing what specific constraint is put on the Fourier power spectrum. We show that it is possible to obliviously sample a Fourier-constrained multidimensional signal with a near-optimal (up to a logarithmic factor) number of samples that guarantee successful reconstruction, partially answering an open question in Avron et al. (2019) which considered the $1$-dimensional case. Our approach highlights a phenomenon that is unique for dimension $d\ge 2$ that the sampling strategy should depend on the geometry of the region on which the signal is to be reconstructed, unlike the case $d=1$ where all regions are of the form $[a,b]$ which are all geometrically equivalent. Our proof, using tools from convex geometry, also illuminates an idea obscured in $d=1$, that to reconstruct a signal in a given region, it can be helpful to take some samples outside that region.

Benign overfitting refers to a recently discovered intriguing phenomenon that over-parameterized neural networks, in many cases, can fit the training data perfectly but still generalize well, surprisingly contrary to the traditional belief that overfitting is harmful for generalization. In spite of its surging popularity in recent years, little has been known in the theoretical aspect of benign overfitting of neural networks. In this work, we provide a theoretical analysis of benign overfitting for two-layer neural networks with possibly non-smooth activation function. Without resorting to the popular Neural Tangent Kernel (NTK) approximation, we prove that neural networks can be trained with gradient descent to classify binary-labeled training data perfectly (achieving zero training loss) even in presence of polluted labels, but still generalize well. Our result removes the smoothness assumption in previous literature and goes beyond the NTK regime; this enables a better theoretical understanding of benign overfitting within a practically more meaningful setting, e.g., with (leaky-)ReLU activation function, small random initialization, and finite network width.

We propose $\textsf{ScaledGD($\lambda$)}$, a preconditioned gradient descent method to tackle the low-rank matrix sensing problem when the true rank is unknown, and when the matrix is possibly ill-conditioned. Using overparametrized factor representations, $\textsf{ScaledGD($\lambda$)}$ starts from a small random initialization, and proceeds by gradient descent with a specific form of preconditioning with a fixed damping term to combat overparameterization. At the expense of light computational overhead incurred by preconditioners, $\textsf{ScaledGD($\lambda$)}$ is remarkably robust to ill-conditioning compared to vanilla gradient descent ($\mathsf{GD}$). Specifically, we show that, under the Gaussian design, $\textsf{ScaledGD($\lambda$)}$ converges to the true low-rank matrix at a constant linear rate that is independent of the condition number (apart from a short nearly dimension-free burdening period), with near-optimal sample complexity. This significantly improves upon the convergence rate of vanilla $\mathsf{GD}$ which suffers from a polynomial dependency with the condition number. Our work provides evidence on the power of preconditioning in accelerating the convergence without hurting generalization in overparameterized learning.

In this letter, we present a differentially private algorithm which accurately estimates the mean of an underlying population with given cumulative distribution function. Our algorithm outperforms the former algorithms in two aspects. First, our algorithm is capably of handling more general types of probability distributions, possibly with a very heavy tail. Second, for light-tailed distributions, our algorithm achieves a better level of accuracy with fewer samples.

Change-point detection (CPD) from streaming data is a fundamental problem in statistical signal processing and has wide applications in wireless sensor networks and the Internet of Things (IoT), such as environmental monitoring and physical activity detection. In various detection schemes, decentralized detection without fusion center is becoming increasingly popular for IoT applications since it enjoys the benefit of strong robustness and security. However, it faces the big challenge of high energy consumption. In this work, we propose an energy-efficient decentralized CPD algorithm called request-response and censoring-based cumulative sum. Specifically, we design a novel communication strategy based on request-response and censoring scheme, which could help the sensors extract useful information from the neighbor sets with a low amount of communication. Furthermore, to provide a guideline on selecting the parameters involved in our algorithm, we theoretically analyze the performance of the proposed algorithm under a simplification for the most interesting cases, in terms of two standard criteria average running length (ARL) and expected detection delay (EDD). Numerical simulations are conducted to verify the effectiveness of the proposed algorithm. Moreover, we test the feasibility of the proposed algorithm in the physical activity CPD task. The experimental results on the real-world data set demonstrate its high energy efficiency and small detection delay.

Dual-function radar communication (DFRC) systems implement both sensing and communication using the same hardware. An emerging DFRC strategy embeds transmission of digital messages into agility-based radar schemes in the form of index modulation (IM). This approach provides the ability to communicate without entailing degradation in radar performance, at the cost of increased decoding complexity at the receiver side. In this work we propose schemes for reducing the decoding complexity associated with IM-based DFRC systems. We first focus on the receiver side, developing a sub-optimal low complexity scheme for recovering IM symbols embedded in radar waveforms. Then, we propose a method to modify the radar waveform to facilitate the recovery of the communicated bits with minimal effect on the radar performance. Our numerical results demonstrate that the proposed techniques allow the receiver to reliably recover the transmitted symbols with an affordable computational burden.

Modern radar systems are expected to operate reliably in congested environments under cost and power constraints. A recent technology for realizing such systems is frequency agile radar (FAR), which transmits narrowband pulses in a frequency hopping manner. To enhance the target recovery performance of FAR in complex electromagnetic environments, and particularly, its range-Doppler recovery performance, multi-Carrier AgilE phaSed Array Radar (CAESAR) was proposed. CAESAR extends FAR to multi-carrier waveforms while introducing the notion of spatial agility. In this paper, we theoretically analyze the range-Doppler recovery capabilities of CAESAR. Particularly, we derive conditions which guarantee accurate reconstruction of these range-Doppler parameters. These conditions indicate that by increasing the number of frequencies transmitted in each pulse, CAESAR improves performance over conventional FAR, especially in complex environments where some radar measurements are severely corrupted by interference.

Detecting subspace signals is an important problem in radar and sonar signal processing, hyperspectral image processing, wireless communication, and other fields. Among these problems, a typical scenario is that one needs to detect a signal lying in a given target subspace, contaminated by interferences that also lie in some subspaces. Most classical works in this aspect treated only the cases where the interfering subspace is known a priori, but in practice the interfering subspace is often unknown. Recently, the arise of Volume Correlation Detector allows to detect subspace signals when the interfering subspace is unknown, but it does not work well in the case where the target subspace overlaps with the interfering subspace. In this paper, we propose a new detector, called Principal Angle Detector (PAD), based on principal angles between subspaces. Our new detector is robust against overlapping target subspace and interfering subspace, and against some other model mismatches. To provide a correct and reasonable theoretical result, we borrow from the literature of numerical linear algebra the tool of affine-invariant covariance estimation, which could be of potential use for related problems in signal processing.