In this paper, we present a coding framework for deep convolutional neural network compression. Our approach utilizes the classical coding theories and formulates the compression of deep convolutional neural networks as a rate-distortion optimization problem. We incorporate three coding ingredients in the coding framework, including bit allocation, dead zone quantization, and Tunstall coding, to improve the rate-distortion frontier without noticeable system-level overhead introduced. Experimental results show that our approach achieves state-of-the-art results on various deep convolutional neural networks and obtains considerable speedup on two deep learning accelerators. Specifically, our approach achieves 20× compression ratio on ResNet-18, ResNet-34, and ResNet-50, and 10× compression ratio on the compact already model MobileNet-v2, without hurting the accuracy. We then examine the system level impact of our approach when deploying the compressed models to hardware platforms. Hardware simulation results show that our approach obtains up to 4.3× and 2.8× inference speedup on state-of-the-art deep learning accelerators TPU and Eyeriss, respectively.

Visual text information is a descriptive part of many images that can be used to perform mobile visual search (MVS) with particularly small queries. In this paper, we propose a system that uses word patch descriptors for retrieving images containing visual text. A random sampling method is used to find duplicate word patches in the database and reduce the database size. The system achieves comparable retrieval performance to state-of-the-art image feature-based systems for images of book covers, and performs better than state-of-the-art text-based retrieval systems for images of book pages. Using visual text to provide distinctive features, our system achieves more than 10-to-1 query size reduction for images of book covers and more than 16-to-1 query size reduction for images of book pages.

MPEG is currently developing a standard titled Compact Descriptors for Visual Search (CDVS) for descriptor extraction and compression. In this work, we report comprehensive patch-level experiments for a direct comparison of low bitrate descriptors for visual search. For evaluating different compression schemes, we propose a dataset of matching pairs of image patches from the MPEG-CDVS image-level data sets. We propose a greedy rate allocation scheme for distributing bits across different spatialbins of the SIFT descriptor. We study a scheme based on Entropy Constrained Vector Quantization and greedy rate allocation, which performs close to the performancebound for any compression scheme. Finally, we present extensive feature-level Receiver Operating Characteristic (ROC) comparisons for different compression schemes (VectorQuantization, Transform Coding, Lattice Coding) proposed during the MPEG-CDVS standardization process.

For mobile augmented reality, an image captured by a mobile device's camera is often compared against a database hosted on a remote server to recognize objects in the image. It is critically important that the amount of data transmitted over the network is as small as possible to reduce the system latency. A low bitrate global signature for still images has been previously shown to achieve high-accuracy image retrieval. In this paper, we develop new methods for interframe coding of a continuous stream of global signatures that can reduce the bitrate by nearly two orders of magnitude compared to independent coding of these global signatures, while achieving the same or better image retrieval accuracy. The global signatures are constructed in an embedded data structure that offers rate scalability. The usage of these new coding methods and the embedded data structure allows the streaming of high-quality global signatures at a bitrate that is less than 2 kbps. Furthermore, a statistical analysis of the retrieval and coding performance is performed to understand the trade off between bitrate and image retrieval accuracy and explain why interframe coding of global signatures substantially outperforms independent coding.

In this paper, they address a key challenge for scaling image search up to larger databases: the amount of memory consumed by the inverted index. In a VT-based image retrieval system, the most memory-intensive structure is the inverted index. For example, in a database of one million images where each image contains hundreds of features, the inverted index consumes 2.5 GB of RAM. Such large memory usage limits the ability to run other concurrent processes on the same server, such as recognition systems for other databases. A memory-congested server can exhibit swapping between main and virtual memory, which significantly slows down all processes.

For mobile image matching applications, a mobile device captures a query image, extracts descriptive features, and transmits these features wirelessly to a server. The server recognizes the query image by comparing the extracted features to its database and returns information associated with the recognition result. For slow links, query feature compression is crucial for low-latency retrieval. Previous image retrieval systems transmit compressed feature descriptors, which is well suited for pairwise image matching. For fast retrieval from large databases, however, scalable vocabulary trees are commonly employed. In this paper, we propose a rate-efficient codec designed for tree-based retrieval. By encoding a tree histogram, our codec can achieve a more than 5times rate reduction compared to sending compressed feature descriptors. By discarding the order amongst a list of features, histogram coding requires 1.5times lower rate than sending a tree node index for every feature. A statistical analysis is performed to study how the entropy of encoded symbols varies with tree depth and the number of features.

Motion-compensated lifted wavelets have received much interest for video compression. While they are biorthogonal, they may substantially deviate from orthonormality due to motion compensation, even if based on an orthogonal or near-orthogonal wavelet. A temporal transform for video sequences that maintains orthonormality while permitting flexible motion compensation would be very desirable. We have recently introduced such a transform for integer-pel accurate motion compensation from one previous frame. In this paper, we extend this idea to half-pel accurate motion compensation. Orthonormality is maintained for arbitrary half-pel motion compensation by cascading a sequence of incremental orthogonal transforms. The half-pel intensity values are obtained by averaging neighboring integer-pel positions. Depending on the number of averaged integer-pel values, we use different types of incremental transforms. The cascade of incremental transforms allows us to choose in each step the optimal type of incremental transform and, hence, the optimal half-pel position. Half-pel motion-compensated blocks of arbitrary shape and size can be used as the granularity of the cascade can be as small as one pixel. The new half-pel accurate motion-compensated orthogonal video transform compares favorably with the integer-pel accurate orthogonal transform

We consider a first-order Markov source, which is predictively encoded using a DPCM-style encoder. The quantized compressed prediction residual is transmitted over an erasure channel. Additionally, a Wyner-Ziv encoded version of the prediction residual is transmitted in order to provide error resilience. When the symbols from the first transmission are erased by the channel, this second description is decoded, and limits the maximum distortion that can occur. Since the quantization step size used in the second description is, in general, larger than that used in the main transmission, error protection is lossy. Using high-rate quantization theory, we derive expressions for the rate and the end-to-end distortion incurred by this system. We show that, compared to conventional lossless forward error correction, this lossy error protection scheme is robust in the sense that it allows the received signal quality to degrade gracefully when the erasure probability increases

Distributed compression is particularly attractive for stereo images since it avoids communication between cameras. Since compression performance depends on exploiting the redundancy between images, knowing the disparity is important at the decoder. Unfortunately, distributed encoders cannot calculate this disparity and communicate it. We consider the compression of grayscale stereo images, and develop an expectation maximization algorithm to perform unsupervised learning of disparity during the decoding procedure. Towards this, we devise a novel method for joint bitplane distributed source coding of grayscale images. Our experiments with both natural and synthetic 8-bit images show that the unsupervised disparity learning algorithm outperforms a system which does no disparity compensation by between 1 and more than 3 bits/pixel and performs nearly as well as a system which knows the disparity through an oracle

We extend the rate-distortion function for Wyner-Ziv coding of noisy sources with quadratic distortion, in the jointly Gaussian case, to more general statistics. It suffices that the noisy observation Z be the sum of a function of the side information Y and independent Gaussian noise, while the source data X must be the sum of a function of Y, a linear function of Z, and a random variable N such that the conditional expectation of N given Y and Z is zero, almost surely. Furthermore, the side information Y may be arbitrarily distributed in any alphabet, discrete or continuous. Under these general conditions, we prove that no rate loss is incurred due to the unavailability of the side information at the encoder. In the noiseless Wyner-Ziv case, i.e., when the source data is directly observed, the assumptions are still less restrictive than those recently established in the literature. We confirm, theoretically and experimentally, the consistency of this analysis with some of the main results on high-rate Wyner-Ziv quantization of noisy sources.

We consider the problem of rate-distortion optimized streaming of packetized media over the Internet from a server to a client using rich acknowledgments. Instead of separately acknowledging each media packet as it arrives, the client periodically sends to the server a single acknowledgment packet, denoted rich acknowledgment, that contains information about all media packets that have arrived at the client by the time the rich acknowledgment is sent. Computing the optimal transmission policy for the server involves estimation of the probability that a single packet will be communicated in error as a function of the expected redundancy (or cost) used to communicate the packet. In this paper, we show how to compute this error-cost function, and thereby optimize the server's transmission policy, in this scenario.

Two separate applications of Wyner-Ziv coding of motion video coding are considered. The Wyner-Ziv theorem on source coding with side information available only at the decoder suggests that an asymmetric video codec, where individual frames are encoded separately, but decoded conditionally could achieve efficiency comparable to current interframe video compression systems. First results on the Wyner-Ziv coding scheme for motion video are reported that use interframe encoding, but interframe decoding. Secondly, Wyner-Ziv coding for the transmission of compressed video over an error-prone channel is applied. This coding scheme is used to generate a supplementary bit stream, which contains a coarsely quantized representation of the transmitted video signal. Using the conventionally decoded, error-concealed video signal as side information to decode the Wyner-Ziv bits, the transmission errors in the decoded video waveform are corrected up to a certain residual distortion, significantly improving the visual quality of the decoded video.

The diversity for media streaming in a rate-distortion optimization framework was considered. A sender-driven transmission scenario was also investigated. Diversity was achieved by using multiple transmission paths over the network. The proposed framework enables the sender to decide at every instant which packets, if any, to transmit and over which transmission paths in order to meet a rate constraint while minimizing the end-to-end distortion. Experimental results demonstrate the benefit of exploiting packet diversity in rate-distortion optimized sender-driven streaming of packetized media.

The problem of designing optimal quantizers for distributed source coding is addressed. The generality of this formulation includes both the symmetric and asymmetric scenarios, together with a number of coding schemes, such as ideal coding achieving a rate equal to the joint conditional entropy of the quantized sources given the side information. The paper shows the optimality conditions that quantizers must satisfy, and generalizes the Lloyd algorithm for their design. Experimental results are shown for the Gaussian scalar asymmetric case.

We show that turbo codes can come close to the Slepian-Wolf bound in lossless distributed source coding. In the asymmetric scenario considered, X and Y are statistically dependent signals and X is encoded with no knowledge of Y. However, Y is known as side information at the decoder. We use a system based on turbo codes to send X at a rate close to H(X|Y). We apply our system to binary sequences and simulations show performance close to the information-theoretic limit. For distributed source coding of Gaussian sequences, our results show significant improvement over previous work. The scheme also performs well for joint source-channel coding.

Multihypothesis motion-compensating predictors combine several motion-compensated signals to predict the current frame of a video signal. This paper applies the wide-sense stationary theory of multihypothesis motion compensation for hybrid video codecs to multihypothesis motion estimation. This allows us to study the influence of the displacement error correlation on the efficiency of multihypothesis motion compensation. Reducing the displacement error correlation between the hypotheses decreases the variance of the multihypothesis prediction error. We derive a property for the displacement error correlation coefficient for an optimal multihypothesis motion estimator in the mean squared error sense. We observe for the wide-sense stationary model that jointly optimal motion estimation improves the prediction performance and reduces the prediction error variance up to 12 dB per accuracy refinement step compared to 6 dB per accuracy refinement step for uncorrelated displacement errors. Consequently, the gain of multihypothesis motion-compensated prediction with jointly optimal motion estimation over motion-compensated prediction increases by improving the accuracy of each hypothesis. We also discuss the combination of hypotheses with additive noise and extend the predictor by the optimum Wiener filter.

Multi-hypothesis motion-compensated prediction extends traditional motion-compensated prediction used in video coding schemes. Known algorithms for block-based multi-hypothesis motion-compensated prediction are, for example, overlapped block motion compensation (OBMC) and bidirectionally predicted frames (B-frames). This paper presents a generalization of these algorithms in a rate-distortion framework. All blocks which are available for prediction are called hypotheses. Further, we explicitly distinguish between the search space and the superposition of hypotheses. Hypotheses are selected from a search space and their spatio-temporal positions are transmitted by means of spatio-temporal displacement codewords. Constant predictor coefficients are used to combine linearly hypotheses of a multi-hypothesis. The presented design algorithm provides an estimation criterion for optimal multi-hypotheses, a rule for optimal displacement codes, and a condition for optimal predictor coefficients. Statistically dependent hypotheses of a multi-hypothesis are determined by an iterative algorithm. Experimental results show that Increasing the number of hypotheses from 1 to 8 provides prediction gains up to 3 dB in prediction error.