Post-training quantization (PTQ) is an effective solution for deploying deep neural networks on edge devices with limited resources. PTQ is especially attractive because it does not require access to the entire original training dataset on the promise of being able to use a much smaller calibration dataset. However, many existing PTQ methods still require a sufficiently large calibration dataset (e.g., more than 1000 images) to achieve satisfactory model accuracy. In this paper, we present a novel post-training quantization method that estimates quantization parameters using a Bayesian Maximum A Posterior (MAP) estimator. By modeling the uncertainty of quantization operations, we formulate the neural network quantization as a Bayesian inference problem. In our method, we first employ probabilistic programming techniques to optimize quantization parameters by maximizing the posterior of quantization step sizes. In addition, we introduce a Minimum Description Length (MDL) prior that favors low quantization bit widths and a validation procedure, which enhances PTQ performance when learning from small calibration datasets. Comprehensive evaluations demonstrate that the proposed method can improve the PTQ performance using a minimal calibration dataset of just 64 images, and achieve nearly state-of-the-art PTQ performance. Furthermore, the proposed method shows strong generalization ability when calibrated on different data sources and tested across diverse data.

With the rapid evolution of deep learning technologies, the efficient deployment of models in real-time and energy-sensitive environments has become increasingly vital. Accurately predicting how deep models consume resources ensures that these models operate within the constraints of their intended scenarios. This is particularly essential for applications such as Neural Architecture Search (NAS), edge computing, and distributed systems. Most current work focuses on predicting layer-wise inference time and energy consumption, but the summed predictions often do not align with the actual runtime and energy cost. This paper introduces a framework using two-stage Multi-Layer Perceptrons (MLPs) to predict model-wise inference time and energy consumption of deep neural networks. The first stage of our approach involves layer-wise inference time and energy usage predictions tailored to various types of neural network layers. Building upon these initial outputs, the second-stage predictor employs another MLP to aggregate these layer-wise estimations into a comprehensive prediction of the overall model’s performance. We validate the effectiveness of the proposed method on two real computing platforms. This framework can enhance the design and deployment of deep learning architectures, by accurately estimating model-wide inference time and energy consumption.

The widespread success of deep learning in solving machine learning problems has fueled its adoption in many fields, from speech recognition to drug discovery and medical imaging. However, deep learning systems are extremely fragile: imperceptibly small modifications to their input data can cause the models to produce erroneous output. It is very easy to generate such adversarial perturbations even for state-of-the-art models, yet immunization against them has proven exceptionally challenging. Despite over a decade of research on this problem, our solutions are still far from satisfactory and many open problems remain. In this work, we survey some of the most important contributions in the field of adversarial robustness. We pay particular attention to the reasons why past attempts at improving robustness have been insufficient, and we identify several promising areas for future research.

Inductive Venn-ABERS predictors (IVAPs) are a type of probabilistic predictors with the theoretical guarantee that their predictions are perfectly calibrated. In this paper, we propose to exploit this calibration property for the detection of adversarial examples in binary classification tasks. By rejecting predictions if the uncertainty of the IVAP is too high, we obtain an algorithm that is both accurate on the original test set and resistant to adversarial examples. This robustness is observed on adversarials for the underlying model as well as adversarials that were generated by taking the IVAP into account. The method appears to offer competitive robustness compared to the state-of-the-art in adversarial defense yet it is computationally much more tractable.

We present Quasar, a new programming framework that handles many complex aspects in the design of smart vision systems on embedded platforms, such as parallelization, data flow management, scheduling and load balancing. Quasar, as a high-level programming language, is nearly hardware-agnostic, has a low barrier of entry and is therefore well suited for algorithm design and rapid prototyping. Through several benchmarks and application use cases we demonstrate that programs written in Quasar have a performance that is on a par with (or better than) hand-tuned CUDA and OpenACC code while the development requires much less time and is future-proof.

A linear, or 1D, camera is a type of camera that sweeps a linear sensor array over the scene, rather than capturing the scene using a single impression on a 2D sensor array. They are often used in satellite imagery, industrial inspection, or hyperspectral imaging. In satellite imaging calibration is often done through a collection of ground points for which the 3D locations are known. In other applications, e.g. hyperspectral imaging, such known points are not available and annotating many different points is onerous. Hence we will use a checkerboard for calibration. The state-of-the-art method for linear camera calibration with a checkerboard becomes unstable when the checkerboards are parallel to the image plane. Our proposed method1yields more accurate camera calibrations without suffering from this shortcoming.

The input-output mappings learned by state-of-the-art neural networks are significantly discontinuous. It is possible to cause a neural network used for image recognition to misclassify its input by applying very specific, hardly perceptible perturbations to the input, called adversarial perturbations. Many hypotheses have been proposed to explain the existence of these peculiar samples as well as several methods to mitigate them. A proven explanation remains elusive, however. In this work, we take steps towards a formal characterization of adversarial perturbations by deriving lower bounds on the magnitudes of perturbations necessary to change the classification of neural networks. The bounds are experimentally verified on the MNIST and CIFAR-10 data sets.

In this letter, we present an efficient image denoising method combining quadtree-based nonlocal means (NLM) and locally adaptive principal component analysis. It exploits nonlocal multiscale self-similarity better, by creating sub-patches of different sizes using quadtree decomposition on each patch. To achieve spatially uniform denoising, we propose a local noise variance estimator combined with denoiser based on locally adaptive principal component analysis. Experimental results demonstrate that our proposed method achieves very competitive denoising performance compared with state-of-the-art denoising methods, even obtaining better visual perception at high noise levels.

Combined detection-estimation tasks are frequently encountered in medical imaging. Optimal methods for joint detection and estimation are of interest because they provide upper bounds on observer performance, and can potentially be utilized for imaging system optimization, evaluation of observer efficiency, and development of image formation algorithms. We present a unified Bayesian framework for decision rules that maximize receiver operating characteristic (ROC)-type summary curves, including ROC, localization ROC (LROC), estimation ROC (EROC), free-response ROC (FROC), alternative free-response ROC (AFROC), and exponentially-transformed FROC (EFROC) curves, succinctly summarizing previous results. The approach relies on an interpretation of ROC-type summary curves as plots of an expected utility versus an expected disutility (or penalty) for signal-present decisions. We propose a general utility structure that is flexible enough to encompass many ROC variants and yet sufficiently constrained to allow derivation of a linear expected utility equation that is similar to that for simple binary detection. We illustrate our theory with an example comparing decision strategies for joint detection-estimation of a known signal with unknown amplitude. In addition, building on insights from our utility framework, we propose new ROC-type summary curves and associated optimal decision rules for joint detection-estimation tasks with an unknown, potentially-multiple, number of signals in each observation.

Multi-camera triangulation of feature points based on a minimisation of the overall ℓ2reprojection error can get stuck in suboptimal local minima or require slow global optimisation. For this reason, researchers have proposed optimising the ℓ∞norm of the ℓ2single view reprojection errors, which avoids the problem of local minima entirely. In this paper we present a novel method for ℓ∞triangulation that minimizes the ℓ∞norm of the ℓ∞reprojection errors: this apparently small difference leads to a much faster but equally accurate solution which is related to the MLE under the assumption of uniform noise. The proposed method adopts a new optimisation strategy based on solving simple quadratic equations. This stands in contrast with the fastest existing methods, which solve a sequence of more complex auxiliary Linear Programming or Second Order Cone Problems. The proposed algorithm performs well: for triangulation, it achieves the same accuracy as existing techniques while executing faster and being straightforward to implement.

In two-view stereo matching, the disparity of occluded pixels cannot accurately be estimated directly: it needs to be inferred through, e.g., regularisation. When capturing scenes using a plenoptic camera or a camera dolly on a track, more than two input images are available, and - contrary to the two-view case - pixels in the central view will only very rarely be occluded in all of the other views. By explicitly handling occlusions, we can limit the depth estimation of pixel p to only use those cameras that actually observe p. We do this by extending variational stereo matching to multiple views, and by explicitly handling occlusion on a view-by-view basis. Resulting depth maps are illustrated to be sharper and less noisy than typical recent techniques working on light fields.

Non-local self-similarity based image denoising depends strictly on similarity measure. The denoising performance is determined based on the ability to reliably find sufficient number of similar patches. In this paper, we propose a rotation invariant similarity measure to fully exploit the image non-local self-similarity. Instead of using image patches, we employ local frequency descriptors, that are rotation invariant and robust to noise, to measure the similarity. Thus, both translational and rotational similarity can be handled even at high noise level. The comparative experimental results show that the proposed method is effective as a rotation invariant similarity measure, and it can consistently improve the performance of non-local means algorithm to achieve better denoising results.

In image and video processing research, rapid prototyping and testing of different variations of an algorithm is quite essential (e.g., to find out if a given algorithm can solve a given problem or work in real-time). In the past decade, the computational performance of graphical processing units (GPUs) has improved significantly, where speed-up factors of 10×-50× compared to single-threaded CPU execution are not uncommon. However, GPU programming is challenging, requiring a significant programming expertise and moreover, most existing programming approaches are not well suited for rapid prototyping. In this Show & Tell session, we present a new programming framework, aimed at making the bridge between high-level program specification and low-level implementation and optimization on heterogeneous computation devices. The goal is that the programmer is relieved from (most) implementation issues and can focus on the specification and improvement of the algorithms. We present a new prototype domain-specific programming language (in the first place aimed at image and video processing) that provides a uniform programming approach toward different hardware devices, a run-time environment to manage and communicate with the heterogeneous devices and an integrated development environment (IDE). The IDE provides various useful image processing-related debugging and profiling features.

Wavelet-based demosaicing techniques have the advantage of being computationally relatively fast, while having a reconstruction performance that is similar to state-of-the-art techniques. Because the demosaicing rules are linear, it is fairly simple to integrate denoising into the demosaicing. In this paper, we present a method that performs joint denoising and demosaicing, using a Gaussian Scale Mixture (GSM) prior model, thereby modeling the local edge direction as a hidden variable. The results indicate that this technique offers a better reconstruction performance (in PSNR sense and visually) than sequential demosaicing and denoising. On a recent GPU, our algorithm takes 3.5 s for reconstructing a 12 megapixel RAW digital camera image.

In the field of image denoising, the non-local means (NLMS) filter is a conceptually simple, yet powerful technique. This filter exploits non-local, i.e. spatially repetitive, structure in natural images to estimate noise-free structure. In contrast, a wide variety of image restoration problems have been solved exploiting local smoothness of natural images, e.g. by enforcing sparsity of images when subjected to a multi-resolution transform. In this paper we introduce the prior knowledge of non-local repetitiveness of image structures into a broad multi-resolution image restoration framework. The proposed framework allows the power of the NLMS filter, supplemented by multi-resolution sparsity, to be extended for a wide variety of image restoration problems, such as demosaicing, deconvolution, reconstruction from insufficient measurements,... in a conceptually simple way.

In this paper, we present a first-order primal-dual algorithm for tackling the joint demosaicking and deconvolution problem. The proposed algorithm exploits the sparsity of both discrete gradient (TV) and shearlet coefficients as prior knowledge. In order to deal with this sparsity across the color channels, we first decorrelate the signals in color space before sparsifying them spatially, resulting in a separable transform. We demonstrate that this approach yields better results than employing group sparsity strategies. We propose to update the decorrelation operator during the image reconstruction, this approach will result in a significant improvement in PSNR. By relaxing the sparsity of the chrominance signals, we obtain both better objective and subjective image quality compared to other state-of-the-art demosaicking and deconvolution algorithms. Also, color artifacts due to demosaicking are suppressed very well.

In this paper, we present a new method for High Dynamic Range (HDR) reconstruction based on a set of multiple photographs with different exposure times. While most existing techniques take a deterministic approach by assuming that the acquired low dynamic range (LDR) images are noise-free, we explicitly model the photon arrival process by assuming sensor data corrupted by Poisson noise. Taking the noise characteristics of the sensor data into account leads to a more robust way to estimate the non-parametric camera response function (CRF) compared to existing techniques. To further improve the HDR reconstruction, we adopt the split-Bregman framework and use Total Variation for regularization. Experimental results on real camera images and ground-truth data show the effectiveness of the proposed approach.

Parallel Imaging MRI (pMRI) and Compressive Sensing (CS) are two reconstruction techniques that have recently been applied to increase MRI performance. In this paper we demonstrate that a combined analysis of the pMRI and CS problems leads to a conceptually simple, yet effective technique that outperforms independent approaches to both reconstruction problems. We argue that the proposed technique is also naturally resilient to noise, due to its relation to the MAP image denoising formulation. A modified Basis Pursuit (BP) formulation of the CS-MRI problem allows it to handle the pMRI problem at the same time. We also present an exact solution to this BP problem, using the split Bregman technique, with discrete shearlet transform (DST) regularization. The DST is an excellent choice for natural image applications, due to its optimal sparsity property. Results show that this Compressive Parallel Sensing (COMPASS) reconstruction algorithm outperforms more traditional MRI reconstruction algorithms in both pMRI and CS experiments.

Fuzzy clustering techniques have been widely used in automated image segmentation. However, since the standard fuzzy c-means (FCM) clustering algorithm does not consider any spatial information, it is highly sensitive to noise. In this paper, we present an extension of the FCM algorithm to overcome this drawback, by incorporating spatial neighborhood information into a new similarity measure. We consider that spatial information depends on the relative location and features of the neighboring pixels. The performance of the proposed algorithm is tested on synthetic and real images with different noise levels. Experimental quantitative and qualitative segmentation results show that the proposed method is effective, more robust to noise and preserves the homogeneity of the regions better than other FCM-based methods.

In this paper, we derive a novel robust image alignment technique that performs joint geometric and photometric registration in the total least square sense. The main idea is to use the total least square metrics instead of the ordinary least square metrics, which is commonly used in the literature. While the OLS model indicates that the target image may contain noise and the reference image should be noise-free, this puts a severe limitation on practical registration problems. By introducing the TLS model, which allows perturbations in both images, we can obtain mutually consistent parameters. Experimental results show that our method is indeed much more consistent and accurate in presence of noise compared to existing registration algorithms.

The dual-tree complex wavelet transform (DT-CWT) uses approximate Hilbert transform pairs of wavelet, which requires that the filters of each tree of the dual-tree structure should be delayed approximately one half sample from each other. However, the filters of the first (finest) scale of the transform do not obey this condition, resulting in a poor directional selectivity for the first scale. In this paper, we describe a design technique for first-scale infinite impulse response (IIR) wavelet filters, that solves this problem. Results demonstrate that a much better directional selectivity is obtained, which indicates a performance improvement for many applications that use the DT-CWT where the preservation of high-frequency information is important.

In high dynamic range (HDR) imaging, multiple photographs with different exposure times are combined into a radiance map, which reflects the radiance in real-life scenes. This involves recovering the response function of the imaging process. The technique proposed by Debevec and Malik is a well-known HDR image synthesis algorithm, but the computational complexity is relatively high, which limits the possible image size and the reconstruction quality. In this paper we present an improved joint optimization technique for estimating the camera response function (CRF) and the radiance map and a new sequential two-step optimization technique, which first estimates the CRF and then reconstructs the radiance map, resulting in better visual results and a higher SNR in remarkably less computation time.

Current clinical practice is increasingly moving in the direction of volumetric imaging. However, model observers for 3D images have been little explored so far. This study is investigating the task of detecting 2D signals in multi-slice simulated image data. We propose a novel design of a multi-slice model observer. To evaluate it, we compare three different model designs of the channelized Hotelling observer (CHO), two multi-slice and one single-slice model. The multi-slice models are built as a sequence of a 2D CHO and 1D HO, where the CHO is used to calculate a vector of metrics for each slice in the planar view and the HO is used to calculate the final scalar statistic of the model. The single-slice model is a 2D CHO applied on the location of the lesion. Our results show that the multi-slice models outperform the single-slice one, and here the new model surpasses the existing one.

Images, captured with digital imaging devices, often contain noise. In literature, many algorithms exist for the removal of white uncorrelated noise, but they usually fail when applied to images with correlated noise. In this paper, we design a new denoising method for the removal of correlated noise, by modeling the significance of the noise-free wavelet coefficients in a local window using a new significance measure that defines the "signal of interest" and that is applicable to correlated noise. We combine the intrascale model with a hidden Markov tree model to capture the interscale dependencies between the wavelet coefficients. We propose a denoising method based on the combined model and a less redundant wavelet transform. We present results that show that the new method performs as well as the state-of-the-art wavelet-based methods, while having a lower computational complexity.

We propose a new statistical model for image restoration in which neighborhoods of wavelet subbands are modeled by a discrete mixture of linear projected Gaussian Scale Mixtures (MPGSM). In each projection, a lower dimensional approximation of the local neighborhood is obtained, thereby modeling the strongest correlations in that neighborhood. The model is a generalization of the recently developed Mixture of GSM (MGSM) model, that offers a significant improvement both in PSNR and visually compared to the current state-of-the-art wavelet techniques. However, the computation cost is very high which hampers its use for practical purposes. We present a fast EM algorithm that takes advantage of the projection bases to speed up the algorithm. The results show that, when projecting on a fixed data-independent basis, even computational advantages with a limited loss of PSNR can be obtained with respect to the BLS-GSM denoising method, while data-dependent bases of Principle Components offer a higher denoising performance, both visually and in PSNR compared to the current wavelet-based state-of-the-art denoising methods.

In image denoising applications, noise is often correlated and the noise energy and correlation structure may even vary with the position in the image. Existing noise reduction and estimation methods are usually designed for stationary white Gaussian noise and generally work less efficient in this case because of the noise model mismatch. In this paper, we propose an EM algorithm for the estimation of spatially variant (nonstationary) correlated image noise in the wavelet domain. In particular, we study additive white Gaussian noise filtered by a space-variant linear filter. This general noise model is applicable to a wide variety of practical situations, including noise in Computed Tomography (CT). Results demonstrate the effectiveness of the proposed solution and its robustness to signal structures.

We develop a new vector-based shrinkage rule, based on the concept of "signal of interest", for the removal of correlated noise. The multivariate Bessel K Form density is used for modeling the spatial correlations between complex wavelet coefficients. The interscale dependencies between the coefficients are captured using a Hidden Markov Tree model. The combined spatial and interscale model gives improvements over recently proposed Hidden Markov Models for white noise. The results show that correlated noise is suppressed well while image details are being preserved.

We develop a low-complexity overcomplete wavelet domain method for denoising digital images corrupted with non-stationary white additive Gaussian noise. The noise level for each pixel is estimated from a local window around that pixel. We use a shrinkage function that adapts itself to the noise level and to the spatially changing statistics of the image. Experiments show that this noise model has good results for different non-stationary noise sources. Finally, we extend our method for denoising images corrupted with signal-dependent noise.