Early time classification algorithms aim to label a stream of features without processing the full input stream, while maintaining accuracy comparable to that achieved by applying the classifier to the entire input. In this paper, we introduce a statistical framework that can be applied to any sequential classifier, formulating a calibrated stopping rule. This data-driven rule attains finite-sample, distribution-free control of the accuracy gap between full and early-time classification. We start by presenting a novel method that builds on the Learn-then-Test calibration framework to control this gap marginally, on average over i.i.d. instances. As this algorithm tends to yield an excessively high accuracy gap for early halt times, our main contribution is the proposal of a framework that controls a stronger notion of error, where the accuracy gap is controlled conditionally on the accumulated halt times. Numerical experiments demonstrate the effectiveness, applicability, and usefulness of our method. We show that our proposed early stopping mechanism reduces up to 94% of timesteps used for classification while achieving rigorous accuracy gap control.

The pursuit of high perceptual quality in image restoration has driven the development of revolutionary generative models, capable of producing results often visually indistinguishable from real data. However, as their perceptual quality continues to improve, these models also exhibit a growing tendency to generate hallucinations – realistic-looking details that do not exist in the ground truth images. Hallucinations in these models create uncertainty about their reliability, raising major concerns about their practical application. This paper investigates this phenomenon through the lens of information theory, revealing a fundamental tradeoff between uncertainty and perception. We rigorously analyze the relationship between these two factors, proving that the global minimal uncertainty in generative models grows in tandem with perception. In particular, we define the inherent uncertainty of the restoration problem and show that attaining perfect perceptual quality entails at least twice this uncertainty. Additionally, we establish a relation between distortion, uncertainty and perception, through which we prove the aforementioned uncertainly-perception tradeoff induces the well-known perception-distortion tradeoff. We demonstrate our theoretical findings through experiments with super-resolution and inpainting algorithms. This work uncovers fundamental limitations of generative models in achieving both high perceptual quality and reliable predictions for image restoration. Thus, we aim to raise awareness among practitioners about this inherent tradeoff, empowering them to make informed decisions and potentially prioritize safety over perceptual performance.

Temporal action segmentation relates to high-level video understanding, commonly formulated as frame-wise classification of untrimmed videos into predefined actions. Fully-supervised deep-learning approaches require dense video annotations which are time and money consuming. Furthermore, the temporal boundaries between consecutive actions typically are not well-defined, leading to inherent ambiguity and interrater disagreement. A promising approach to remedy these limitations is timestamp supervision, requiring only one labeled frame per action instance in a training video. In this work, we reformulate the task of temporal segmentation as a graph segmentation problem with weakly-labeled vertices. We introduce an efficient segmentation method based on random walks on graphs, obtained by solving a sparse system of linear equations. Furthermore, the proposed technique can be employed in any one or combination of the following forms: (1) as a standalone solution for generating dense pseudo-labels from timestamps; (2) as a training loss; (3) as a smoothing mechanism given intermediate predictions. Extensive experiments with three datasets (50Salads, Breakfast, GTEA) show that our method competes with state-of-the-art, and allows the identification of regions of uncertainty around action boundaries.

In recent years there has been increasing interest in leveraging denoisers for solving general inverse problems. Two leading frameworks are regularization-by-denoising (RED) and plug-and-play priors (PnP) which incorporate explicit likelihood functions with priors induced by denoising algorithms. RED and PnP have shown state-of-the-art performance in diverse imaging tasks when powerful denoisersare used, such as convolutional neural networks (CNNs). However, the study of their convergence remains an active line of research. Recent works derive the convergence of RED and PnP methods by treating CNN denoisers as approximations for maximum a posteriori (MAP) or minimum mean square error (MMSE) estimators. Yet, state-of-the-art denoisers cannot be interpreted as either MAPor MMSE estimators, since they typically do not exhibit symmetric Jacobians. Furthermore, obtaining stable inverse algorithms often requires controlling the Lipschitz constant of CNN denoisers during training. Precisely enforcing this constraint is impractical, hence, convergence cannot be completely guaranteed. In this work, we introduce image denoisers derived as the gradients of smooth scalar-valued deep neural networks, acting as potentials. This ensures two things: (1) the proposed denoisers display symmetric Jacobians, allowing for MAP and MMSE estimators interpretation; (2) the denoisers may be integrated into RED and PnP schemes with backtracking step size, removing the need for enforcing their Lipschitz constant. To show the latter, we develop a simple inversion method that utilizes the proposed denoisers. We theoretically establish its convergence to stationary points of an underlying objective function consisting of the learned potentials. We numerically validate our method through various imaging experiments, showing improved results compared to standard RED and PnP methods, and with additional provable stability.

Inverse problems in image processing are typically cast as optimization tasks, consisting of data fidelity and stabilizing regularization terms. A recent regularization strategy of great interest utilizes the power of denoising engines. Two such methods are the plug-and-play prior (PnP) and regularization by denoising (RED). While both have shown state-of-the-art results in various recovery tasks, their theoretical justification is incomplete. In this paper, we aim to bridge RED and PnP, enriching the understanding of both frameworks. Toward that end, we reformulate RED as a convex optimization problem utilizing a projection (RED-PRO) onto the fixed-point set of demicontractive denoisers. We offer a simple iterative solution to this problem, by which we show that under certain conditions the PnP proximal gradient method is a special case of RED-PRO, while providing guarantees for the convergence of both frameworks to globally optimal solutions. In addition, we present relaxations of RED-PRO that allow for handling denoisers with limited fixed-point sets. Finally, we demonstrate RED-PRO for the tasks of image deblurring and superresolution, showing improved results with respect to the original RED framework.

In this article, we consider deep learning strategies in ultrasound systems, from the front end to advanced applications. Our goal is to provide the reader with a broad understanding of the possible impact of deep learning methodologies on many aspects of ultrasound imaging. In particular, we discuss methods that lie at the interface of signal acquisition and machine learning, exploiting both data structure (e.g., sparsity in some domain) and data dimensionality (big data) already at the raw radio-frequency channel stage. As some examples, we outline efficient and effective deep learning solutions for adaptive beamforming and adaptive spectral Doppler through artificial agents, learn compressive encodings for the color Doppler, and provide a framework for structured signal recovery by learning fast approximations of iterative minimization problems, with applications to clutter suppression and super-resolution ultrasound. These emerging technologies may have a considerable impact on ultrasound imaging, showing promise across key components in the receive processing chain.

Biomedical imaging is unequivocally dependent on the ability to reconstruct interpretable and high-quality images from acquired sensor data. This reconstruction process is pivotal across many applications, spanning from magnetic resonance imaging to ultrasound imaging. While advanced data-adaptive reconstruction methods can recover much higher image quality than traditional approaches, their implementation often poses a high computational burden. In ultrasound imaging, this burden is significant, especially when striving for low-cost systems, and has motivated the development of high-resolution and high-contrast adaptive beamforming methods. Here we show that deep neural networks, that adopt the algorithmic structure and constraints of adaptive signal processing techniques, can efficiently learn to perform fast high-quality ultrasound beamforming using very little training data. We apply our technique to two distinct ultrasound acquisition strategies (plane wave, and synthetic aperture), and demonstrate that high image quality can be maintained when measuring at low data-rates, using undersampled array designs. Beyond biomedical imaging, we expect that the proposed deep learning based adaptive processing framework can benefit a variety of array and signal processing applications, in particular when data-efficiency and robustness are of importance.

Contrast enhanced ultrasound is a radiation-free imaging modality which uses encapsulated gas microbubbles for improved visualization of the vascular bed deep within the tissue. It has recently been used to enable imaging with unprecedented subwavelength spatial resolution by relying on super-resolution techniques. A typical preprocessing step in super-resolution ultrasound is to separate the microbubble signal from the cluttering tissue signal. This step has a crucial impact on the final image quality. Here, we propose a new approach to clutter removal based on robust principle component analysis (PCA) and deep learning. We begin by modeling the acquired contrast enhanced ultrasound signal as a combination of low rank and sparse components. This model is used in robust PCA and was previously suggested in the context of ultrasound Doppler processing and dynamic magnetic resonance imaging. We then illustrate that an iterative algorithm based on this model exhibits improved separation of microbubble signal from the tissue signal over commonly practiced methods. Next, we apply the concept of deep unfolding to suggest a deep network architecture tailored to our clutter filtering problem which exhibits improved convergence speed and accuracy with respect to its iterative counterpart. We compare the performance of the suggested deep network on both simulations and in-vivo rat brain scans, with a commonly practiced deep-network architecture and with the fast iterative shrinkage algorithm. We show that our architecture exhibits better image quality and contrast.

Sparse arrays are of great interest since they can identify O(N2) uncorrelated sources with N physical sensors. This stems from their large difference coarray, defined as the differences between sensor locations. In a recent study, desired array properties such as closed-form expression for sensor locations, symmetry and large hole-free difference coarray were considered and it was shown that most existing sparse arrays do not exhibit these characteristics simultaneously. Standard Cantor arrays were shown to satisfy all the criteria above, however, their difference coarrays are of size O(Nlog2 3) which is smaller than that obtained with minimum redundancy arrays and nested arrays. In this paper, we introduce a fractal array design where a generator array is extended in a simple recursive fashion. In contrast to previous work, the generator is assumed to be a sparse array with a hole-free difference coarray. We study the resulting arrays and prove they inherit their properties from the generator. Thus, this approach can be used to extend any known sparse configuration to an arbitrarily large array. A small-scale array, which meets all design criteria, can be created and then expanded to generate a symmetric fractal array with a difference coarray of size O(N2), unlike Cantor arrays.

Sparse and low-rank decomposition, also known as robust principle component analysis, has been applied successfully in numerous applications. Typically, this approach leads to a minimization problem which is solved using iterative algorithms. Drawing inspiration from recurrent networks, in recent years deep-learning strategies have been extended to mimic the behavior of iterative algorithms, with reduced complexity. In this work, we propose an extension of these deep architectures to robust principle component analysis in which fully-connected layers are replaced with convolutional ones. This strategy offers spatial invariance and significant reduction in the number of learned parameters. We then apply the proposed method to contrast-enhanced ultrasound, in which low-rank tissue signal needs to be removed in order to visualize blood vessels. We demonstrate the effectiveness of our approach on simulations and in-vivo rat brain scans. The resulting images exhibit improved visual quality and contrast compared with images obtained by commonly practiced methods.

The real-time nature that makes diagnostic ultrasonography so appealing to clinicians imposes strong constraints on the computational complexity of image reconstruction algorithms. As such, these typically rely on traditional delay-and-sum beamforming, a low-complexity approach that unfortunately comes at the cost of reduced image quality as compared to more advanced and content-adaptive beamformers. Here, we propose a model-aware deep learning strategy to ultrasound image reconstruction, which leverages knowledge of minimum variance beamforming while exploiting the efficiency of deep neural networks. Our approach yields high quality images with strong contrast at real-time reconstruction rates. The neural network is trained using in vivo and simulated radio frequency channel data of a single plane wave transmit, and corresponding high-quality minimum-variance beamformed reconstructions. Performance is benchmarked using simulated acquisitions from the PICMUS [1] dataset, demonstrating the convincing generalizability and image quality of the proposed beamformer.

The number of elements of a uniform linear array (ULA) is the main bottleneck in many applications in array processing in terms of cost and power consumption. This motivates the use of sparse arrays where some of the elements are removed. However, designing a sparse array configuration with the smallest number of elements that preserves the full array beam pattern is generally NP hard. In this paper we adopt work in multiple-input-multiple-output (MIMO) radar to study a sparse array composed of two sub apertures. We derive the minimal number of elements required using this design, showing that in general there are two optimal solutions. Next, we present an extension of this approach beyond two sub apertures. By optimizing the number of sub apertures, we prove that the optimal array configuration is related to the notion of prime factorization. This allows to achieve a significant reduction in the number of elements.