Standard video codecs are rate-distortion optimization machines, where distortion is typically quantified using PSNR versus the source. However, it is now widely accepted that increasing PSNR does not necessarily translate to better visual quality. In this paper, a better balance between perception and fidelity is targeted, in order to provide for significant rate savings over state-of-the-art standards-based video codecs. Specifically, pre- and post-processing neural networks are proposed that enhance the coding efficiency of standard video codecs when bench-marked with an array of well-established perceptual quality scores. These "neural wrapper" elements are end-to-end trained with a neural codec module serving as a differentiable proxy for standard video codecs. The codec proxy is jointly optimized with the pre- and post components via a novel two-phase pretraining strategy and end-to-end iterative refinement with stop-gradient. This allows the neural pre- and postprocessor to learn to embed, remove and recover information in a codec-aware manner, thus improving its rate-quality performance. A single neural-wrapper model is thereby established and used for the entire rate-quality curve without needing any downscaling or upscaling. The trained model is tested with the AV1 and VVC standard codecs via an array of well-established objective quality scores (SSIM, MS-SSIM, VMAF, AVQT), as well as mean opinion scores (MOS) derived from ITU-T P.910 subjective testing. Experimental results show that the proposed approach improves all quality scores, with -18.5% average Bjontegaard Delta-rate (BD-rate) saving over all objective scores and MOS improvement over both standard codecs. This illustrates the significant potential of neural wrapper components over standards-based video coding.

Video processing algorithms like video upscaling, denoising, and compression are now increasingly optimized for perceptual quality metrics instead of signal distortion. This means that they may score well for metrics like video multi-method assessment fusion (VMAF), but this may be because of metric overfitting. This imposes the need for costly subjective quality assessments that cannot scale to large datasets and large parameter explorations. We propose a methodology that fuses multiple quality metrics based on small scale subjective testing in order to unlock their use at scale for specific application domains of interest. This is achieved by employing pseudo-random sampling of the resolution, quality range and test video content available, which is initially guided by quality metrics in order to cover the quality range useful to each application. The selected samples then undergo a subjective test, such as ITU-T P.910 absolute categorical rating, with the results of the test postprocessed and used as the means to derive the best combination of multiple objective metrics using support vector regression. We showcase the benefits of this approach in two applications: video encoding with and without perceptual preprocessing, and deep video denoising & upscaling of compressed content. For both applications, the derived fusion of metrics allows for a more robust alignment to mean opinion scores than a perceptually-uninformed combination of the original metrics themselves. The dataset and code is available at https://github.com/isize-tech/VideoQualityFusion.

Anticipation of accidents ahead of time in autonomous and non -autonomous vehicles aids in accident avoidance. In order to recognize abnormal events such as traffic accidents in a video sequence, it is important that the network takes into account interactions of objects in a given frame. We propose a novel Feature Aggregation (FA) block that refines each object's features by computing a weighted sum of the features of all objects in a frame. We use FA block along with Long Short Term Memory (LSTM) network to anticipate accidents in the video sequences. We report mean Average Precision (mAP) and Average Time-to-Accident (ATTA) on Street Accident (SA) dataset. Our proposed method achieves the highest score for risk anticipation by predicting accidents 0.32 sec and 0.75 sec earlier compared to the best results with Adaptive Loss and dynamic parameter prediction based methods respectively.

Neural Architecture Search (NAS) has gained attraction due to superior classification performance. Differential Architecture Search (DARTS) is a computationally light method. To limit computational resources DARTS makes numerous approximations. These approximations result in inferior performance. We propose to fine-tune DARTS using fixed operations as they are independent of these approximations. Our method offers a good trade-off between the number of parameters and classification accuracy. Our approach improves the top-1 accuracy on Fashion-MNIST, CompCars, and MIO-TCD datasets by 0.56%, 0.50%, and 0.39%, respectively compared to the state-of-the-art approaches. Our approach performs better than DARTS, improving the accuracy by 0.28%, 1.64%, 0.34%, 4.5%, and 3.27% compared to DARTS, on CIFAR-10, CIFAR-100, Fashion-MNIST, CompCars, and MIO-TCD datasets, respectively.

Network compression reduces the computational complexity and memory consumption of deep neural networks by reducing the number of parameters. In SVD-based network compression the right rank needs to be decided for every layer of the network. In this paper we propose an efficient method for obtaining the rank configuration of the whole network. Unlike previous methods which consider each layer separately, our method considers the whole network to choose the right rank configuration. We propose novel accuracy metrics to represent the accuracy and complexity relationship for a given neural network. We use these metrics in a non-iterative fashion to obtain the right rank configuration which satisfies the constraints on FLOPs and memory while maintaining sufficient accuracy. Experiments show that our method provides better compromise between accuracy and computational complexity/memory consumption while performing compression at much higher speed. For VGG-16 our network can reduce the FLOPs by 25% and improve accuracy by 0.7% compared to the baseline, while requiring only 3 minutes on a CPU to search for the right rank configuration. Previously, similar results were achieved in 4 hours with 8 GPUs. The proposed method can be used for lossless compression of a neural network as well. The better accuracy and complexity compromise, as well as the extremely fast speed of our method make it suitable for neural network compression.

Numerous camera sensors have been recently proposed to estimate the depth of objects in the scene. However, many of these sensors either require multiple imaging elements or an active illuminant. In this paper, we propose a new depth camera, termed offset aperture (OA) camera, which provides depth of the scene with a single shot without an active illuminant. The camera is based on a 4-color sensor that senses the infrared (IR) signal along with the visible RGB signals. Separate OAs are used for IR and RGB images. A depth-dependent disparity is generated between the IR and RGB images. There is no disparity across the R, G, and B channel images, which allows the RGB image to be well-aligned. Numerous techniques are used in this work to observe good-quality depth maps and images. Leakage compensation has been proposed to remove crosstalk between the IR and RGB images. A comparison with another 4-color sensor called Dual Aperture camera shows that OA camera provides 88% improvement in depth sensitivity and 45% improvement in color quality, on average. The proposed OA camera can be used for numerous applications including occlusion handling in stereo, selective focus, and night vision. The advantages of the proposed OA camera include competitive image quality, capability of depth extraction, well-aligned RGB images, and small footprint of the camera, which make OA camera a suitable candidate for mobile applications.

Crowd anomaly detection is a key research area in vision-based surveillance. Most of the crowd anomaly detection algorithms are either too slow, bulky, or power-hungry to be applicable for battery-powered surveillance cameras. In this paper, we present a new crowd anomaly detection algorithm. The proposed algorithm creates a feature for every superpixel that includes the contribution from the neighboring superpixels only if their direction of motion conforms with the dominant direction of motion in the region. We also propose using univariate Gaussian discriminant analysis with the K-means algorithm for classification. Our method provides superior accuracy over numerous deep learning-based and handcrafted feature-based approaches. We also present a low-power FPGA implementation of the proposed method. The algorithm is developed such that features are extracted over non-overlapping pixels. This allows gating inputs to numerous modules resulting in higher power efficiency. The maximum energy required per pixel is 2.43 nJ in our implementation. 126.65 Mpixels can be processed per second by the proposed implementation. The speed, power, and accuracy performance of our method make it competitive for surveillance applications, especially battery-powered surveillance cameras.

This paper presents a Offset Aperture (OA) based single camera system and proposes a optimized vision processor, a new hardware architecture for fast, low-energy, and low-complexity depth extraction. The proposed design was fabricated in 110nm CMOS image sensor technology and supports 32-level depth resolution on 1920×1080 full HD image with 30fps, consuming 280.53mW from 1.5V supply and a mere 2.8% of bad classification. The low-complexity algorithms are employed to eliminate the DRAM access, thereby the proposed OA architecture can be directly embedded with the CMOS image sensor and commercial image processing chip.

Due to the increasing demand for 3D applications, development of novel depth-sensing cameras is being actively pursued. However, most of these cameras still face the challenge of high energy consumption and slow speed in the depth extraction process. This becomes a serious bottleneck in embedded implementations where real-time performance is required, constrained by power and area. This work proposes Offset Aperture (OA) camera, a new hardware architecture for fast, low-energy, and low-complexity depth extraction. Optimal implementations of pre-processing, cost-volume generation and cost-aggregation are presented. The whole depth-extraction pipeline has been implemented on a Field Programmable Gate Array (FPGA). Overall, a mere 2.8% of bad classification was achieved with the proposed system. Also, the proposed system can process 37 VGA frames per second while consuming 0.224 μJ/pixel. High accuracy, speed and low energy consumption of the proposed OA architecture make it suitable for embedded applications.

Pedestrian detection is a key problem in computer vision and is currently addressed with increasingly complex solutions involving compute-intensive features and classification schemes. In this scope, histogram of oriented gradients (HOG) in conjunction with linear support vector machine (SVM) classifier is considered to be the single most discriminative feature that has been adopted as a stand-alone detector as well as a key instrument in advance systems involving hybrid features and cascaded detectors. In this paper, we propose a pedestrian detection framework that is computationally less expensive as well as more accurate than HOG-linear SVM. The proposed scheme exploits the discriminating power of the locally significant gradients in building orientation histograms without involving complex floating point operations while computing the feature. The integer-only feature allows the use of powerful histogram inter-section kernel SVM classifier in a fast lookup-table-based implementation. Resultantly, the proposed framework achieves at least 3% more accurate detection results than HOG on standard data sets while being 1.8 and 2.6 times faster on conventional desktop PC and embedded ARM platforms, respectively, for a single scale pedestrian detection on VGA resolution video. In addition, hardware implementation on Altera Cyclone IV field-programmable gate array results in more than 40% savings in logic resources compared with its HOG-linear SVM competitor. Hence, the proposed feature and classification setup is shown to be a better candidate as the single most discriminative pedestrian detector than the currently accepted HOG-linear SVM.

Background subtraction (BS) is a crucial machine vision scheme for detecting moving objects in a scene. With the advent of smart cameras, the embedded implementation of BS finds ever-increasing applications. This paper presents a new BS scheme called efficient BS for smart cameras (EBSCam). EBSCam thresholds the change in the estimated background model, which suppresses variance of the estimates, resulting in competitive performance compared with standard BS schemes. The percentage of wrong classification of EBSCam is lower than those of the Gaussian mixture model (GMM) (10.97%) and the pixel-based adaptive segmenter (PBAS) (4.66%) algorithms in FPGA implementations. Moreover, the memory bandwidth requirement of EBSCam is 6.66%, 41.36%, and 90.48% lower than the state-of-the-art FPGA implementation of GMM, ViBe, and PBAS algorithms, respectively. EBSCam achieves a significant speed up compared with the FPGA implementations of GMM (by 43.3%), ViBe (by 118.6%), and PBAS (by 144.8%) schemes. Similarly, the energy consumption of EBSCam is 80.56% and 99.9% less compared with GMM and PBAS, respectively. In summary, the advantages of EBSCam in accuracy, speed, and energy consumption combined together make it especially suitable for embedded applications.

Pedestrian detection has lately attracted considerable interest from researchers due to many practical applications. However, the low accuracy and high complexity of pedestrian detection has still not enabled its use in successful commercial applications. In this paper, we present insights into the complexity-accuracy relationship of pedestrian detection. We consider the Histogram of Oriented Gradients (HOG) scheme with linear Support Vector Machine (LinSVM) as a benchmark. We describe parallel implementations of various blocks of the pedestrian detection system which are designed for full-HD (1920×1080) resolution. Features are improved by optimal selection of cell size and histogram bins which have been shown to significantly affect the accuracy and complexity of pedestrian detection. It is seen that with a careful choice of these parameters a frame rate of 39.2 fps is achieved with a negligible loss in accuracy which is 16.3x and 3.8x higher than state of the art GPU and FPGA implementations respectively. Moreover 97.14% and 10.2% reduction in energy consumption is observed to process one frame. Finally, features are further enhanced by removing petty gradients in histograms which result in loss of accuracy. This increases the frame rate to 42.7 fps (18x and 4.1x higher) and lowers the energy consumption by 97.34% and 16.4% while improving the accuracy by 2% as compared to state of the art GPU and FPGA implementations respectively.

CCTV-based surveillance systems gaining widespread popularity still waste computational power, transmission bandwidth and storage space. This paper tries to respond to this necessity by proposing a motion-based video recording scheme with dual frame rate motion detection. Statistical models for memory and energy consumption of the overall system are described. Root-mean-square error of the models for memory consumption is 0.54 and 0.78 for systems with single frame rate and dual frame rate motion detection, respectively. For a typical surveillance video, the proposed dual frame rate system stores 1.22, 4.94, 7.81 less frames per second on at 10fps, 1fps, 0.5fps respectively, compared to the single frame rate motion detection system. We have suggested a criterion for using dual frame rate motion detection in surveillance based on a mathematical model.

Ultra-low voltage VLSI designs are gaining widespread attention due to the requirements of minimum energy consuming motes. Leakage energy is a considerable part of the overall energy consumed in the ultra-low voltage domain. This paper deals with the overall energy reduction in ultra-low power systems by the use of digit-serial implementations. Leakage energy is reduced by the reduction of hardware in digit-serial implementations. However, overhead circuitry adds to the energy budget. It has been shown in this paper that digit-serial implementations do reduce the overall energy consumption in the ultra-low power VLSI circuits compared to bit-serial and wordparallel implementations. 73% and 92% reduction in energy per clock cycle was obtained with an 8-bit adder at a source voltage of 0.7V compared to the bit-serial and word-parallel implementations respectively.

In this paper surveillance system employing background subtraction scheme is discussed. The aim of the work is to minimize the waste energy of the overall system due to false positives. Pixels in the foreground of a motion detection system remain non-zero even if a moving object has stopped due to the settling time associated with an adaptive background subtraction scheme as Mixture of Gaussians. Temporal variance in a visually static pixel region also triggers false positives. Optimal learning rate for different parameters as threshold values, ROI size and total number of frames in the scene has been derived in this paper.