Video traffic comprises a large majority of the total traffic on the internet today. Uncompressed visual data requires a very large data rate; lossy compression techniques are employed in order to keep the data-rate manageable. Increasingly, a significant amount of visual data being generated is consumed by analytics (such as classification, detection, etc.) residing in the cloud. Image and video compression can produce visual artifacts, especially at lower data-rates, which can result in a significant drop in performance on such analytic tasks. Moreover, standard image and video compression techniques aim to optimize perceptual quality for human consumption by allocating more bits to perceptually significant features of the scene. However, these features may not necessarily be the most suitable ones for semantic tasks. We present here an approach to compress visual data in order to maximize performance on a given analytic task. We train a deep auto-encoder using a multi-task loss to learn the relevant embeddings. An approximate differentiable model of the quantizer is used during training which helps boost the accuracy during inference. We apply our approach on an image classification problem and show that for a given level of compression, it achieves higher classification accuracy than that obtained by performing classification on images compressed using JPEG. Our approach also outperforms the relevant state-of-the-art approach by a significant margin.

We develop a Deep Learning-based Wearable Vision-system with Vibrotactile-feedback (DLWV2)to guide Blind and Visually Impaired (BVI)people to reach objects. The system achieves high accuracy in object detection and tracking in 3-D using an extended deep learning-based 2.5-D detector and a 3-D object tracker with the ability to track 3-D object locations even outside the camera field-of-view. We train our detector with a large number of images with 2.5-D object ground-truth (i.e., 2-D object bounding boxes and distance from the camera to objects). A novel combination of HTC Vive Tracker with our system enables us to automatically obtain the ground-truth labels for training while requiring very little human effort to set up the system. Moreover, our system processes frames in real-time through a client-server computing platform such that BVI people can receive realtime vibrotactile guidance. We conduct a thorough user study on 12 BVI people in new environments with object instances which are unseen during training. Our system outperforms the non-assistive guiding strategy with statistic significance in both time and the number of contacting irrelevant objects. Finally, the interview with BVI users confirms that our system with distance-based vibrotactile feedback is mostly preferred, especially for objects requiring gentle manipulation such as a bottle with water inside.

Human pose-estimation in a multi-person image involves detection of various body parts and grouping them into individual person clusters. While the former task is challenging due to mutual occlusions, the combinatorial complexity of the latter task is very high. We propose a greedy part assignment algorithm that exploits the inherent structure of the human body to lower the complexity of the graphical model, compared to any of the prior published works. This is accomplished by (i) reducing the number of part-candidates using the estimated number of people in the image, (ii) doing a greedy sequential assignment of partclasses, following the kinematic chain from head to ankle (iii) doing a greedy assignment of parts in each part-class set, to person-clusters (iv) limiting the candidate person clusters to the most proximal clusters using human anthropometric data and (v) using only a specific subset of pre-assigned parts for establishing pairwise structural constraints. We show that, these steps sparsify the bodyparts relationship graph and reduces the algorithm's complexity to be linear in the number of candidates of any single part-class. We also propose a method for spawning person-clusters from any unassigned significant body part to make the algorithm robust to occlusions. We show that, our proposed part-assignment algorithm, despite using a sub-optimal pre-trained DNN model, achieves state of the art results on both MPII and WAF pose datasets, demonstrating the robustness of our approach.

This paper presents a no-reference perceptual metric that quantifies the degree of perceived regularity in textures. The metric is based on the similarity of visual attention (VA) of the textural primitives and the periodic spatial distribution of foveated fixation regions throughout the image. A ground-truth eye-tracking database for textures is also generated as part of this paper and is used to evaluate the performance of the most popular VA models. Using the saliency map generated by the best VA model, the proposed texture regularity metric is computed. It is shown through subjective testing that the proposed metric has a strong correlation with the mean opinion score for the perceived regularity of textures. The proposed texture regularity metric can be used to improve the quality and performance of many image processing applications like texture synthesis, texture compression, and content-based image retrieval.

This paper presents a reduced-reference quality metric that quantifies the perceptual quality of the synthesized textures. The metric is based on the change in perceived regularity between the original and the synthesized textures. The perceived regularity is quantified through a modified texture regularity metric based on visual attention. It is shown through subjective testing that the proposed metric has a strong correlation with the Mean Opinion Score for the fidelity of synthesized textures and outperforms the state-of-the-art full-reference quality metrics.

The Canny edge detector is one of the most widely used edge detection algorithms due to its superior performance. Unfortunately, not only is it computationally more intensive as compared with other edge detection algorithms, but it also has a higher latency because it is based on frame-level statistics. In this paper, we propose a mechanism to implement the Canny algorithm at the block level without any loss in edge detection performance compared with the original frame-level Canny algorithm. Directly applying the original Canny algorithm at the block-level leads to excessive edges in smooth regions and to loss of significant edges in high-detailed regions since the original Canny computes the high and low thresholds based on the frame-level statistics. To solve this problem, we present a distributed Canny edge detection algorithm that adaptively computes the edge detection thresholds based on the block type and the local distribution of the gradients in the image block. In addition, the new algorithm uses a nonuniform gradient magnitude histogram to compute block-based hysteresis thresholds. The resulting block-based algorithm has a significantly reduced latency and can be easily integrated with other block-based image codecs. It is capable of supporting fast edge detection of images and videos with high resolutions, including full-HD since the latency is now a function of the block size instead of the frame size. In addition, quantitative conformance evaluations and subjective tests show that the edge detection performance of the proposed algorithm is better than the original frame-based algorithm, especially when noise is present in the images. Finally, this algorithm is implemented using a 32 computing engine architecture and is synthesized on the Xilinx Virtex-5 FPGA. The synthesized architecture takes only 0.721 ms (including the SRAM READ/WRITE time and the computation time) to detect edges of 512 × 512 images in the USC SIPI database when clocked at 100 MHz and is faster than existing FPGA and GPU implementations.

This paper presents a no reference perceptual metric that quantifies the degree of regularity in textures. The metric is based on the probability of visual attention at each pixel of the texture image, similarity of visual attention of the textural primitives and the periodic spatial distribution of foveated fixation regions throughout the image. It is shown through subjective testing that the proposed metric has a strong correlation with the Mean Opinion Score for the regularity of textures.

This paper proposes a distributed Canny edge detection algorithm which can be mapped onto multi-core architectures for high throughput applications. In contrast to the conventional Canny edge detection algorithm which makes use of the global image gradient histogram to determine the threshold for edge detection, the proposed algorithm adaptively computes the edge detection threshold based on the local distribution of the gradients in the considered image block. The efficacy of the distributed Canny in detecting psycho-visually important edges is validated using a visual sharpness metric. The proposed distributed Canny edge detection algorithm has the capacity to scale up the throughput adaptively, based on the number of computing engines. The algorithm achieves about 72 times speed up for a 16-core architecture, without any change in performance. Furthermore, the internal memory requirements are significantly reduced especially for smaller block sizes. For instance, if a 512×512 image is processed in 64×64 blocks using the proposed scheme, the memory is reduced by a factor of 70 as compared to the original Canny edge detector.

This paper presents a novel scheme for extracting a still background occluded by a number of foreground objects, moving in different directions and velocities in a video sequence, such that every background pixel is exposed in at least one of the frames. Each identified foreground object is decomposed into blocks. The proposed scheme is able to efficiently estimate, for each foreground block, a source frame from which the occluded background pixels can be extracted. The pixels of the identified source frames are used to populate the co-located occluded pixels in the initial frame. The efficacy and the simplicity of the algorithm lie in its capacity to recover the background directly from the estimated source frames instead of performing a foreground-background classification for every frame. The proposed algorithm is robust to variations in lighting and is effective in removing both rigid and deformable foreground objects. Simulation results are presented to illustrate the performance of the proposed scheme.

The computation of existing sharpness/blurriness objective metrics involves measuring the spread of edge pixels in blurred images. However, in blurred images, many edges might go undetected causing the metrics to become inaccurate. In these scenarios, proper recovery of edge pixels can lead to a better correlation between the perceived sharpness and the sharpness metric. This paper presents an iterative edge refinement algorithm. The proposed edge refinement scheme is integrated into a perceptual- based no-reference sharpness metric resulting in an increased correlation with the perceived sharpness and, thus, in an increased sharpness/blurriness prediction accuracy. Results are presented to illustrate the performance of the proposed scheme and metric.