Neural collapse provides an elegant mathematical characterization of learned last layer representations (a.k.a. features) and classifier weights in deep classification models. Such results not only provide insights but also motivate new techniques for improving practical deep models. However, most of the existing empirical and theoretical studies in neural collapse focus on the case that the number of classes is small relative to the dimension of the feature space. This paper extends neural collapse to cases where the number of classes are much larger than the dimension of feature space, which broadly occur for language models, retrieval systems, and face recognition applications. We show that the features and classifier exhibit a generalized neural collapse phenomenon, where the minimum one-vs-rest margins is maximized. We provide empirical study to verify the occurrence of generalized neural collapse in practical deep neural networks. Moreover, we provide theoretical study to show that the generalized neural collapse provably occurs under unconstrained feature model with spherical constraint, under certain technical conditions on feature dimension and number of classes.

Consider the fundamental problem of arranging r-dimensional subspaces of Rdin such a way that maximizes the minimum distance between unit vectors in different subspaces. It is well known that equi-isoclinic tight fusion frames (EITFFs) are optimal for this packing problem, but such ensembles are notoriously hard to construct. In this paper, we present a novel construction of EITFFs that are totally symmetric: any permutation of the subspaces can be realized by an orthogonal transformation of ℝd.

For $d\in\{5,6\}$, we classify arrangements of $d + 2$ points in ${RP}^{d-1}$ for which the minimum distance is as large as possible. To do so, we leverage ideas from matrix and convex analysis to determine the best possible codes that contain equiangular lines, and we introduce a notion of approximate Positivstellensatz certificates that promotes numerical approximations of Stengle's Positivstellensatz certificates to honest certificates.

Many clustering problems can be solved using semidefinite programming. Theoretical results in this vein frequently consider data with a planted clustering and a notion of signal strength such that the semidefinite program exactly recovers the planted clustering when the signal strength is sufficiently large. In practice, semidefinite programs are notoriously slow, and so speedups are welcome. In this paper, we show how to sketch a popular semidefinite relaxation of a graph clustering problem known as minimum bisection, and our analysis supports a meta-claim that the clustering task is less computationally burdensome when there is more signal.

Given labeled points in a high-dimensional vector space, we seek a low-dimensional subspace such that projecting onto this subspace maintains some prescribed distance between points of differing labels. Intended applications include compressive classification. Taking inspiration from large margin nearest neighbor classification, this paper introduces a semidefinite relaxation of this problem. Unlike its predecessors, this relaxation is amenable to theoretical analysis, allowing us to provably recover a planted projection operator from the data.

We introduce a model-free, parameter-free relax-and-round algorithm for k-means clustering, based on a semidefinite programming relaxation (SDP) due to Peng and Wei [1]. The algorithm interprets the SDP output as a denoised version of the original data and then rounds this output to a hard clustering. We analyze the performance of this algorithm in the setting where the data is drawn from a subgaussian mixture model. We also study the fundamental limits of estimating subgaussian centers with k-means clustering in order to compare our approximation guarantee to the theoretically optimal k-means clustering solution. In particular, our guarantee has no dependence on the number of points, and for equidistant clusters with O(k) separation, our guarantee is optimal up to a factor of k.

An equiangular tight frame (ETF) is a set of equal norm vectors in a Euclidean space whose coherence is as small as possible, equaling the Welch bound. Also known as Welch-bound-equality sequences, such frames arise in various applications, such as waveform design, quantum information theory, compressed sensing, and algebraic coding theory. ETFs seem to be rare, and only a few methods of constructing them are known. In this paper, we present a new infinite family of complex ETFs that arises from hyperovals in finite projective planes. In particular, we give the first ever construction of a complex ETF of 76 vectors in a space of dimension 19. Recently, a computer-assisted approach was used to show that real ETFs of this size do not exist, resolving a longstanding open problem in this field. Our construction is a modification of a previously known technique for constructing ETFs from balanced incomplete block designs.

We consider the classical 1D phase retrieval problem. In order to overcome the difficulties associated with phase retrieval from measurements of the Fourier magnitude, we treat recovery from the magnitude of the short-time Fourier transform (STFT). We first show that the redundancy offered by the STFT enables unique recovery for arbitrary nonvanishing inputs, under mild conditions. An efficient algorithm for recovery of a sparse input from the STFT magnitude is then suggested, based on an adaptation of the recently proposed GESPAR algorithm. We demonstrate through simulations that using the STFT leads to improved performance over recovery from the oversampled Fourier magnitude with the same number of measurements.

Hyperspectral data is commonly used by astronomers to discern the chemical composition of stars. Unfortunately, conventional hyperspectral platforms require long exposure times, which can hamper their use in applications like celestial navigation. We propose a compressed sensing platform that exploits the spatial sparsity of stars to quickly sample the hyperspectral data. We leverage certain combinatorial designs to devise coded apertures, and then we apply block orthogonal matching pursuit to quickly reconstruct the desired imagery.

In many areas of imaging science, it is difficult to measure the phase of linear measurements. As such, one often wishes to reconstruct a signal from intensity measurements, that is, perform phase retrieval. In this paper, we provide a novel measurement design which is inspired by interferometry and exploits certain properties of expander graphs. We also give an efficient phase retrieval procedure, and use recent results in spectral graph theory to produce a stable performance guarantee which rivals the guarantee for PhaseLift in [Candès, Strohmer, and Voroninski, PhaseLift: Exact and Stable Signal Recovery from Magnitude Measurements via Convex Programming, preprint, arXiv:1109.4499, 2011]. We use numerical simulations to illustrate the performance of our phase retrieval procedure, and we compare reconstruction error and runtime with a common alternating-projections-type procedure.

We propose a new mathematical and algorithmic framework for unsupervised image segmentation, which is a critical step in a wide variety of image processing applications. We have found that most existing segmentation methods are not successful on histopathology images, which prompted us to investigate segmentation of a broader class of images, namely those without clear edges between the regions to be segmented. We model these images as occlusions of random images, which we call textures, and show that local histograms are a useful tool for segmenting them. Based on our theoretical results, we describe a flexible segmentation framework that draws on existing work on nonnegative matrix factorization and image deconvolution. Results on synthetic texture mosaics and real histology images show the promise of the method.

An equiangular tight frame (ETF) is a set of unit vectors in a Euclidean space whose coherence is as small as possible, equaling the Welch bound. Also known as Welch-bound-equality sequences, such frames arise in various applications, such as waveform design and compressed sensing. At the moment, there are only two known flexible methods for constructing ETFs: harmonic ETFs are formed by carefully extracting rows from a discrete Fourier transform; Steiner ETFs arise from a tensor-like combination of a combinatorial design and a regular simplex. These two classes seem very different: the vectors in harmonic ETFs have constant amplitude, whereas Steiner ETFs are extremely sparse. We show that they are actually intimately connected: a large class of Steiner ETFs can be unitarily transformed into constant-amplitude frames, dubbed Kirkman ETFs. Moreover, we show that an important class of harmonic ETFs is a subset of an important class of Kirkman ETFs. This connection informs the discussion of both types of frames: some Steiner ETFs can be transformed into constant-amplitude waveforms making them more useful in waveform design; some harmonic ETFs have low spark, making them less desirable for compressed sensing. We conclude by showing that real-valued constant-amplitude ETFs are equivalent to binary codes that achieve the Grey-Rankin bound, and then construct such codes using Kirkman ETFs.

This paper is concerned with an important matrix condition in compressed sensing known as the restricted isometry property (RIP). We demonstrate that testing whether a matrix satisfies RIP isNP-hard. As a consequence of our result, it is impossible to efficiently test for RIP providedP≠NP.

Digital fingerprinting is a framework for marking media files, such as images, music, or movies, with user-specific signatures to deter illegal distribution. Multiple users can collude to produce a forgery that can potentially overcome a fingerprinting system. This paper proposes an equiangular tight frame fingerprint design which is robust to such collusion attacks. We motivate this design by considering digital fingerprinting in terms of compressed sensing. The attack is modeled as linear averaging of multiple marked copies before adding a Gaussian noise vector. The content owner can then determine guilt by exploiting correlation between each user's fingerprint and the forged copy. The worst case error probability of this detection scheme is analyzed and bounded. Simulation results demonstrate that the average-case performance is similar to the performance of orthogonal and simplex fingerprint designs, while accommodating several times as many users.

We show that equiangular tight frames (ETFs) are particularly well suited as additive fingerprint designs against Gaussian averaging collusion attacks when the number of users is less than the square of the signal dimension. The detector performs a binary hypothesis test in order to decide whether a user of interest is among the colluders. Given a maximum coalition size, we show that the geometric figure of merit of distance between the corresponding "guilty" and "not guilty" linear forgeries for each user is bounded away from zero. Moreover, we show that for a normalized correlation detector, reliable detection is guaranteed provided that the number of users is less than the square of the signal dimension. Moreover, we show that the coalition has the best chance of evading detection when it uses equal weights.

The sparse signal processing literature often uses random sensing matrices to obtain performance guarantees. Unfortunately, in the real world, sensing matrices do not always come from random processes. It is therefore desirable to evaluate whether an arbitrary matrix, or frame, is suitable for sensing sparse signals. To this end, the present paper investigates two parameters that measure the coherence of a frame: worst-case and average coherence. We first provide several examples of frames that have small spectral norm, worst-case coherence, and average coherence. Next, we present a new lower bound on worst-case coherence and compare it to the Welch bound. Later, we propose an algorithm that decreases the average coherence of a frame without changing its spectral norm or worst-case coherence. Finally, we use worst-case and average coherence, as opposed to the Restricted Isometry Property, to garner near-optimal probabilistic guarantees on both sparse signal detection and reconstruction in the presence of noise. This contrasts with recent results that only guarantee noiseless signal recovery from arbitrary frames, and which further assume independence across the nonzero entries of the signal-in a sense, requiring small average coherence replaces the need for such an assumption.