Adversarial attack methods can induce machine learning classifiers to mislabel errors. Current methods pay much attention to errors in the image space, i.e. the imperceptibility of adversarial perturbations, to avoid attacks being detected by humans. However, they overlook errors in the label space, i.e. the similarity between the wrong label and the true label. It is easy for humans to detect attacks if the wrong label has a big difference with the true label, for example, a dog is mislabeled as a cat. In this paper, we propose a novel attack method called LabelFool which attacks images with undetectable errors in both label space and image space. Given a classifier, for each input image, LabelFool first predicts the true label by estimating its probability distribution, then selects one label perceptually nearest to the predicted true label as the target label. Then LabelFool generates the adversarial sample by moving the input image towards the classification boundary between the predicted true label and the target label. The subjective experiments on ImageNet and visual results on CASIA-WebFace show that LabelFool is less detectable in the label space than other attack methods. Moreover, LabelFool has low perceptibility in the image space together with a high attack rate.

Transmission electron tomography (TET) is a widely used biomedical imaging technique. Iterative reconstruction with regularization is the common approach to obtain high-quality reconstructed images for TET. Mumford-Shah (MS) regularization has the advantage in preserving image edges; but it is computation and memory-intensive because it is NP-hard when applied to TET. In this work, we design an FPGA accelerator for iterative image reconstruction for TET with the MS regularization. We first design the accelerator with multiple processing elements (PE) to leverage the parallelism in the TET reconstruction. We then schedule the forward projection and back projection according to the imaging geometry of TET to optimize off-chip memory access. Finally, we optimize the local buffer with customized partitioning for the stencil memory access of basic operations to increase the throughput of a single PE. Our FPGA accelerator achieves 6.68X and 1.87X speedup with the same image quality compared with the same algorithm implemented on CPU and GPU, respectively. And the off-chip bandwidth requirements of forward and back projection are reduced by one to two orders of magnitude. Compared with the state-of-the-art acceleration works on GPU and FPGA, we achieve 1.18X and 1.91X throughput, respectively.

This companion paper supports the experimental replication of the paper "Norm-in-Norm Loss with Faster Convergence and Better Performance for Image Quality Assessment'' presented at ACM Multimedia 2020. We provide the software package for replicating the implementation of the "Norm-in-Norm'' loss and the corresponding "LinearityIQA'' model used in the original paper. This paper contains the guidelines to reproduce all the experimental results of the original paper.

X-ray computed tomography (CT) is commonly used to obtain vivo images to characterize diseases but results in radiation exposure to patients. Low-dose CT (LDCT) provides CT images of clinical quality with reduced cumulative radiation dose. Iterative image reconstruction methods with effective regularization are used for LDCT but generally require more computing resources and induce higher computational load than the conventional filtered backprojection (FBP) methods. The high computational demand of the iterative reconstruction (IR) with notably increased reconstruction time precludes its routine clinical application. In this work, we focus on the FPGA acceleration of a compute-intensive full IR (full-IR) algorithm based on the Mumford-Shah regularization. At the algorithmic level, we propose a beam-based asynchronous update algorithm to reduce the computational cost and alleviate the conflicts. At the hardware-level, we first present pipeline-friendly optimization for the original algorithm to increase the computation throughput. We then apply the LDCT-specific tiling strategy to improve the data reuse rate. The experimental results show that our implementation takes 8.5 min to reconstruct a typical physical phantom with the image quality comparable with the vendor's result. The FPGA implementation achieves 11.6× throughput against the state-of-the-art GPU version.

FPGAs are becoming significant for implementing low-latency convolutional neural networks, because of performance demands and power constraints. Conventional implementations of convolutional layers are usually direct convolution, involving nested loops over channels, feature maps, and filters. Explicit general matrix multiplications (GEMMs) cost extra memory space, and the limited on-chip RAMs prevent an efficient GEMM-based implementation. In this paper, we evaluate a low-memory method of GEMMs on FPGAs based systolic arrays. We design a novel accelerator to save the bandwidth and increase the parallelism. We evaluate our design on MobileNet V1 and Inception V4. Our implementation achieves a throughput of around 3.5 TOP/s for both models. We also reduce the memory usage by 21% compared to explicit GEMM implementation for MobileNet V1 and 44% for Inception V4.

In this work, we aim to accelerate the iterative reconstruction (IR) algorithm used for 3D low dose computer tomography (LDCT) reconstruction to reduce the long execution time from an order of several hours on CPU to a few minutes. IR algorithms such as Mumford-Shah (MS) regularization can be used to get high-quality images even though the signal-noise ratio (SNR) of low dose projection data is low. However, IR is a computation and memory-intensive application and the long execution time precludes its clinical application. We adopt the ray-based parallel algorithm and designed a customized processing engine with multiple parallel processing elements (PEs) on field-programmable gate array (FPGA) to improve the computation efficiency. To reduce resource utilization, we proposed a best-first search algorithm combined with pruning to find the optimal bit width for fixed-point reconstruction. Besides, an offline memory optimization framework based on a greedy based clustering algorithm is proposed to reduce external memory bandwidth requirement and balance the workload of parallel PEs. Experiments on a 3D Shepp-Logan phantom show 2.81X and 1.91X speedup over the state of art single GPU and FPGA implementation.

Currently, most image quality assessment (IQA) models are supervised by the MAE or MSE loss with empirically slow convergence. It is well-known that normalization can facilitate fast convergence. Therefore, we explore normalization in the design of loss functions for IQA. Specifically, we first normalize the predicted quality scores and the corresponding subjective quality scores. Then, the loss is defined based on the norm of the differences between these normalized values. The resulting "Norm-in-Norm" loss encourages the IQA model to make linear predictions with respect to subjective quality scores. After training, the least squares regression is applied to determine the linear mapping from the predicted quality to the subjective quality. It is shown that the new loss is closely connected with two common IQA performance criteria (PLCC and RMSE). Through theoretical analysis, it is proved that the embedded normalization makes the gradients of the loss function more stable and more predictable, which is conducive to the faster convergence of the IQA model. Furthermore, to experimentally verify the effectiveness of the proposed loss, it is applied to solve a challenging problem: quality assessment of in-the-wild images. Experiments on two relevant datasets (KonIQ-10k and CLIVE) show that, compared to MAE or MSE loss, the new loss enables the IQA model to converge about 10 times faster and the final model achieves better performance. The proposed model also achieves state-of-the-art prediction performance on this challenging problem. For reproducible scientific research, our code is publicly available at \urlhttps://github.com/lidq92/LinearityIQA.

Blind image deblurring is a challenging problem in computer vision, which aims to restore both the blur kernel and the latent sharp image from only a blurry observation. Inspired by the prevalent self-example prior in image super-resolution, in this paper, we observe that a coarse enough image down-sampled from a blurry observation is approximately a low-resolution version of the latent sharp image. We prove this phenomenon theoretically and define the coarse enough image as a latent structure prior of the unknown sharp image. Starting from this prior, we propose to restore sharp images from the coarsest scale to the finest scale on a blurry image pyramid and progressively update the prior image using the newly restored sharp image. These coarse-to-fine priors are referred to as multi-scale latent structures (MSLSs). Leveraging the MSLS prior, our algorithm comprises two phases: 1) we first preliminarily restore sharp images in the coarse scales and 2) we then apply a refinement process in the finest scale to obtain the final deblurred image. In each scale, to achieve lower computational complexity, we alternately perform a sharp image reconstruction with fast local self-example matching, an accelerated kernel estimation with error compensation, and a fast non-blind image deblurring, instead of computing any computationally expensive non-convex priors. We further extend the proposed algorithm to solve more challenging non-uniform blind image deblurring problem. The extensive experiments demonstrate that our algorithm achieves the competitive results against the state-of-the-art methods with much faster running speed.

Transposed convolution, which is often used to scale up feature maps in various computer vision tasks, is a structural inverse process of convolution. Both convolution and transposed convolution, if any, account for the majority of computation in the inferences of deep neural networks. While convolution has been studied extensively, there are few investigations on accelerating transposed convolution. In this paper, we propose a fast algorithm, FTConv, to reduce the computation of transposed convolution using the Winograd algorithm, which has also been used for convolution with small kernels. Specifically, a transposed convolution can be converted into multiple convolutions after dividing the kernel into several congruence classes. Thus, we can accelerate the multiple convolutions using a modified Winograd algorithm. The transposed convolution can be obtained by interleaving output feature elements of each congruence class. We also design a Winograd ALU in four pipeline stages to further accelerate the computation on FPGA. By carefully designing a sliding window for on-chip buffer reuse according to the memory access pattern of transposed convolution, we save the memory bandwidth by 88.2% compared with a straightforward method. We evaluate FTConv using FSRCNN-s, a neural network for super-resolution. The number of multiplications in the transposed convolution layer can be reduced by 69% over the direct computation of FSRCNN-s.

Most existing image quality assessment models focus on evaluating the image quality score, however, the quality score alone is not enough to characterize the degeneration. In this paper, we propose a full reference framework named Mask Gated Convolutional Network (MGCN) for evaluating the image quality score and identifying distortions simultaneously. Observing the fact that the reference images are distorted by various distortions in pixel space, we design an encoder module to capture the transformation between reference images and distorted images as low level features. Further higher level features are extracted from the low level features and shared by both the regression and the classification tasks. Instead of simply cropping patches to augment data, we mask the high level feature map in the spatial domain to randomly sample patches from the image and learn to assign the image quality score to the patch set. The proposed method achieves the state-of-the-art performance on LIVE2, TID2008 and TID2013 datasets.

Quality assessment of in-the-wild videos is a challenging problem because of the absence of reference videos and shooting distortions. Knowledge of the human visual system can help establish methods for objective quality assessment of in-the-wild videos. In this work, we show two eminent effects of the human visual system, namely, content-dependency and temporal-memory effects, could be used for this purpose. We propose an objective no-reference video quality assessment method by integrating both effects into a deep neural network. For content-dependency, we extract features from a pre-trained image classification neural network for its inherent content-aware property. For temporal-memory effects, long-term dependencies, especially the temporal hysteresis, are integrated into the network with a gated recurrent unit and a subjectively-inspired temporal pooling layer. To validate the performance of our method, experiments are conducted on three publicly available in-the-wild video quality assessment databases: KoNViD-1k, CVD2014, and LIVE-Qualcomm, respectively. Experimental results demonstrate that our proposed method outperforms five state-of-the-art methods by a large margin, specifically, 12.39%, 15.71%, 15.45%, and 18.09% overall performance improvements over the second-best method VBLIINDS, in terms of SROCC, KROCC, PLCC and RMSE, respectively. Moreover, the ablation study verifies the crucial role of both the content-aware features and the modeling of temporal-memory effects. The PyTorch implementation of our method is released at https://github.com/lidq92/VSFA.

Image content variation is a typical and challenging problem in no-reference image-quality assessment (NR-IQA). This work pays special attention to the impact of image content variation on NR-IQA methods. To better analyze this impact, we focus on blur-dominated distortions to exclude the impacts of distortion-type variations. We empirically show that current NR-IQA methods are inconsistent with human visual perception when predicting the relative quality of image pairs with different image contents. In view of deep semantic features of pretrained image classification neural networks always containing discriminative image content information, we put forward a new NR-IQA method based on semantic feature aggregation (SFA) to alleviate the impact of image content variation. Specifically, instead of resizing the image, we first crop multiple overlapping patches over the entire distorted image to avoid introducing geometric deformations. Then, according to an adaptive layer selection procedure, we extract deep semantic features by leveraging the power of a pretrained image classification model for its inherent content-aware property. After that, the local patch features are aggregated using several statistical structures. Finally, a linear regression model is trained for mapping the aggregated global features to image-quality scores. The proposed method, SFA, is compared with nine representative blur-specific NR-IQA methods, two general-purpose NR-IQA methods, and two extra full-reference IQA methods on Gaussian blur images (with and without Gaussian noise/JPEG compression) and realistic blur images from multiple databases, including LIVE, TID2008, TID2013, MLIVE1, MLIVE2, BID, and CLIVE. Experimental results show that SFA is superior to the state-of-the-art NR methods on all seven databases. It is also verified that deep semantic features play a crucial role in addressing image content variation, and this provides a new perspective for NR-IQA.

The market benefits from a barrage of Ultra High Definition (Ultra-HD) displays, yet most extant cameras are barely equipped with Full-HD video capturing. In order to upgrade existing videos without extra storage costs, we propose an FPGA-based super-resolution system that enables real-time Ultra-HD upscaling in high quality. Our super-resolution system crops each frame into blocks, measures their total variation values, and dispatches them accordingly to a neural network or an interpolation module for upscaling. This approach balances the FPGA resource utilization, the attainable frame rate, and the image quality. Evaluations demonstrate that the proposed system achieves superior performance in both throughput and reconstruction quality, comparing to current approaches.

Stochastic gradient descent (SGD) is a widely-used algorithm in many applications, especially in the training process of deep learning models. Low-precision implementation for SGD has been studied as a major acceleration approach. However, if not appropriately used, low-precision implementation can deteriorate its convergence because of the rounding error when gradients become small near a local optimum. In this work, to balance throughput and algorithmic accuracy, we apply the Q-learning technique to adjust the precision of SGD automatically by designing an appropriate decision function. The proposed decision function for Q-learning takes the error rate of the objective function, its gradients, and the current precision configuration as the inputs. Q-learning then chooses proper precision adaptively for hardware efficiency and algorithmic accuracy. We use reconfigurable devices such as FPGAs to evaluate the adaptive precision configurations generated by the proposed Q-learning method. We prototype the framework using LeNet-5 model with MNIST and CIFAR10 datasets and implement it on a Xilinx KCU1500 FPGA board. In the experiments, we analyze the throughput of different precision representations and the precision-selection of our framework. The results show that the proposed framework with adapative precision increases the throughput by up to 4.3× compared to the conventional 32-bit floating point setting, and it achieves both the best hardware efficiency and algorithmic accuracy.

Low-dose X-ray computed tomography (XCT) is a popular imaging technique to visualize the inside structure of object non-destructively. Model-based Iterative Reconstruction (MBIR) method can reconstruct high-quality image but at the cost of large computational demands. Therefore, MBIR of ten resorts to the platforms with hardware accelerators such as GPUs to speed up the reconstruction process.

To guarantee a satisfying Quality of Experience (QoE) for consumers, it is required to measure image quality efficiently and reliably. The neglect of the high-level semantic information may result in predicting a clear blue sky as bad quality, which is inconsistent with human perception. Therefore, in this paper, we tackle this problem by exploiting the high-level semantics and propose a novel no-reference image quality assessment method for realistic blur images. Firstly, the whole image is divided into multiple overlapping patches. Secondly, each patch is represented by the high-level feature extracted from the pre-trained deep convolutional neural network model. Thirdly, three different kinds of statistical structures are adopted to aggregate the information from different patches, which mainly contain some common statistics i.e., the mean & standard deviation, quantiles and moments). Finally, the aggregated features are fed into a linear regression model to predict the image quality. Experiments show that, compared with low-level features, high-level features indeed play a more critical role in resolving the aforementioned challenging problem for quality estimation. Besides, the proposed method significantly outperforms the state-of-the-art methods on two realistic blur image databases and achieves comparable performance on two synthetic blur image databases.

X-ray computed tomography is an important technique for clinical diagnose and nondestructive testing. In many applications a number of image processing steps are needed before the image information becomes useful. Image segmentation is one of such processing steps and has important applications. The conventional flow is to first reconstruct the image and then obtain image segmentation afterwards. In contrast, an iterative method for simultaneous reconstruction and segmentation (SRS) with Mumford-Shah model has been proposed, which not only regularizes the ill-posedness of the tomographic reconstruction problem, but also produces the image segmentation at the same time. The Mumford-Shah model is both mathematically and computationally difficult. In this paper, we propose a data-decomposed algorithm of the SRS method, accelerate it using FPGA devices. The proposed algorithm has a structure that invokes a single kernel many times without involving other computational tasks. Though this structure seems best fit on GPU-like devices, experimental results show that a 73X, 11X, and 1.4X speedup can be achieved by the FPGA acceleration over the CPU implementation of the original SRS algorithm and ray-parallel SRS algorithm, and the GPU implementation of the ray-parallel SRS.

Super-resolution enhancement is a kind of promising approach to enhance the spatial resolution of images. To super-resolve a satisfying result, regularization term design and blur kernel estimation are two important aspects which need to be carefully considered. In this paper, we propose a robust regularized super-resolution reconstruction approach based on two sparsity properties to deal with these two aspects. Firstly, we design a sparse reweighted TV L1 prior to restrict the first derivative of the upsampled image. Then, noticing that only deblurring sparse high gradient areas can sharpen the super-resolution result, we design an over-deblurring control method to decrease the artifacts caused by inaccurate blur kernel estimation. We also design a fast optimization algorithm to solve our model. The experimental results show that the proposed approach achieves a remarkable performance both in visual quality and run time.

The conventional approach of computed tomography (CT) is to solve each image processing task individually in sequence. An obvious drawback is that the measured data is only used once at the first step, and the possible errors, from noises in the measured data, inappropriate modeling, or inappropriate parameters, are not easy to be corrected and will be propagated into the later steps. As a consequence, approaches that combine the reconstruction and the specific processing task have become popular. This work adopts an iterative algorithm with simultaneous reconstruction and segmentation using the Mumford-Shah model, which can be applied not only to regularize the ill-posedness of the tomographic reconstruction problem, but also to compute segmentation directly from the measured data. The Mumford-Shah model is both mathematically and computationally difficult. In this paper, we accelerated this computation and data intensive application by FPGA devices and achieved 9.24X speedup over the conventional CPU implementation.

In many imaging applications, the continuous phase information of the measured signal is wrapped to a single period of 2pi, resulting in phase ambiguity. In this paper we consider the two-dimensional phase unwrapping problem and propose a maximum a posteriori (MAP) framework for estimating the true phase values based on the wrapped phase data. In particular, assuming a joint Gaussian prior on the original phase image, we show that the MAP formulation leads to a binary quadratic minimization problem. The latter can be efficiently solved by semidefinite relaxation (SDR). We compare the performances of our proposed method with the existing L1/L2-norm minimization approaches. The numerical results demonstrate that the SDR approach significantly outperforms the existing phase unwrapping methods.

The Landweber scheme is an algebraic reconstruction method and includes several important algorithms as its special cases. The convergence of the Landweber scheme is of both theoretical and practical importance. Using the singular value decomposition (SVD), we derive an iterative representation formula for the Landweber scheme and consequently establish the necessary and sufficient conditions for its convergence. In addition to verifying the necessity and sufficiency of known convergent conditions, we find new convergence conditions allowing relaxation coefficients in an interval not covered by known results. Moreover, it is found that the Landweber scheme can converge within finite iterations when the relaxation coefficients are chosen to be the inverses of squares of the nonzero singular values. Furthermore, the limits of the Landweber scheme in all convergence cases are shown to be the sum of the minimum norm solution of a weighted least-squares problem and an oblique projection of the initial image onto the null space of the system matrix.

Computed tomography (CT) has been extensively studied for years and widely used in the modern society. Although the filtered back-projection algorithm is the method of choice by manufacturers, efforts are being made to revisit iterative methods due to their unique advantages, such as superior performance with incomplete noisy data. In 1984, the simultaneous algebraic reconstruction technique (SART) was developed as a major refinement of the algebraic reconstruction technique (ART). However, the convergence of the SART has never been established since then. In this paper, the convergence is proved under the condition that coefficients of the linear imaging system are nonnegative. It is shown that from any initial guess the sequence generated by the SART converges to a weighted least square solution.

We introduce a general iterative scheme for image reconstruction based on Landweber's method. In our configuration, a sequential block-iterative (SeqBI) version can be readily formulated from a simultaneous block-iterative (SimBI) version, and vice versa. This provides a mechanism to derive new algorithms from known ones. It is shown that some widely used iterative algorithms, such as the algebraic reconstruction technique (ART), simultaneous ART (SART), Cimmino's, and the recently designed diagonal weighting and component averaging algorithms, are special examples of the general scheme. We prove convergence of the general scheme under conditions more general than assumed in earlier studies, for its SeqBI and SimBI versions in the consistent and inconsistent cases, respectively. Our results suggest automatic relaxation strategies for the SeqBI and SimBI versions and characterize the dependence of the limit image on the initial guess. It is found that in all cases the limit is the sum of the minimum norm solution of a weighted least-squares problem and an oblique projection of the initial image onto the null space of the system matrix.

To discriminate fine anatomical features in the inner ear, it has been desirable that spiral computed tomography (CT) may perform beyond their current resolution limits with the aid of digital image processing techniques. In this paper, we develop a blind deblurring approach to enhance image resolution retrospectively without complete knowledge of the underlying point spread function (PSF). An oblique CT image can be approximated as the convolution of an isotropic Gaussian PSF and the actual cross section. Practically, the parameter of the PSF is often unavailable. Hence, estimation of the parameter for the underlying PSF is crucially important for blind image deblurring. Based on the iterative deblurring theory, we formulate an edge-to-noise ratio (ENR) to characterize the image quality change due to deblurring. Our blind deblurring algorithm estimates the parameter of the PSF by maximizing the ENR, and deblurs images. In the phantom studies, the blind deblurring algorithm reduces image blurring by about 24%, according to our blurring residual measure. Also, the blind deblurring algorithm works well in patient studies. After fully automatic blind deblurring, the conspicuity of the submillimeter features of the cochlea is substantially improved.

We generalize the axiomatic quantification of one-dimensional (1-D) image resolution to the multidimensional case. The imaging system of interest is characterized by a nonnegative spatially invariant point spread function. The axioms extended from the 1-D counterparts include nonnegativity, continuity, translation invariance, rotation invariance, luminance invariance, homogeneous scaling, and serial combination properties. It is proved that the only resolution measure consistent with the axioms is proportional to the square root of the trace of the covariance matrix of the point spread function.