Convolutional neural networks (CNNs) are now widely deployed in a variety of computer vision (CV) systems. These systems typically include an image signal processor (ISP), even though the ISP is traditionally designed to produce images that look appealing to humans. In CV systems, it is not clear what the role of the ISP is, or if it is even required at all for accurate prediction. In this work, we investigate the efficacy of the ISP in CNN classification tasks and outline the system-level trade-offs between prediction accuracy and computational cost. To do so, we build software models of a configurable ISP and an imaging sensor to train CNNs on ImageNet with a range of different ISP settings and functionality. Results on ImageNet show that an ISP improves accuracy by 4.6%-12.2% on MobileNets. Results from ResNets demonstrate these trends also generalize to deeper networks. An ablation study of the various processing stages in a typical ISP reveals that the tone mapper is the most significant stage when operating on high dynamic range (HDR) images, by providing 5.8% average accuracy improvement alone. We also show that the ISP increases the generalization of CNNs across two different image sensors by a significant 17.5 %. Overall, the ISP benefits system efficiency because the memory and computational costs of the ISP is minimal compared to the cost of using a larger CNN to achieve the same accuracy.

Along with the rapid increase in the availability of high-quality video formats such as high definition (HD), ultra HD, and high dynamic range, a huge demand for data rates during their transmission has become inevitable. Consequently, the role of video compression techniques has become crucially important in the process of mitigating the data rate requirements. Even though the latest video codec high efficiency video coding (HEVC) has succeeded in significantly reducing the data rate compared to its immediate predecessor H.264/advanced video coding, the HEVC coded videos in the meantime have become even more vulnerable to network impairments. Therefore, it is equally important to assess the consumers' perceived quality degradation prior to transmitting HEVC coded videos over an error-prone network, and to include error resilient features so as to minimize the adverse effects of those impairments. To this end, this paper proposes a probabilistic model which accurately predicts the overall distortion of the decoded video at the encoder followed by an accurate QP-λ relationship which can be used in the rate-distortion optimization (RDO) process. During the derivation process of the probabilistic model, the impacts from the motion vectors, the pixels in the reference frames, and the clipping operations are accounted, and consequently, the model is capable of minimizing the prediction error to as low as 3.11%, whereas the state-of-the-art methods cannot reach below 20.08%, under identical conditions. Furthermore, the enhanced RDO process has resulted in 21.41%-43.59% improvement in the BD-rate compared to the state-of-the-art error resilient algorithms.

Determining the best partitioning structure of a coding tree unit is one of the most time-consuming operations in High Efficiency Video Coding (HEVC) encoding. Specifically, it is the evaluation of the quadtree hierarchy using the rate-distortion (RD) optimization that has the most significant impact on the encoding time, especially in the cases of high definition (HD) and ultra HD videos. In order to expedite the encoding for low-delay applications, this paper proposes a coding unit (CU) size selection and encoding algorithm for inter prediction in the HEVC. To this end, it describes: 1) two CU classification models based on Inter N × N mode motion features and RD cost thresholds to predict the CU split decision; 2) an online training scheme for dynamic content adaptation; 3) a motion vector reuse mechanism to expedite the motion estimation process; and 4) finally introduces a computational complexity to coding efficiency tradeoff process to enable flexible control of the algorithm. The experimental results reveal that the proposed algorithm achieves a consistent average encoding time performance ranging from 55%-58% and 57%-61% with average Bjøntegaard delta bit rate increases of 1.93%-2.26% and 2.14%-2.33% compared with the HEVC 16.0 reference software for the low delay P and low delay B configurations, respectively, across a wide range of content types and bit rates.

The performance of binaural speech source localization systems can be significantly impacted by an imperfect selection of spatial localization cues, due to the limited bandwidth of speech, and the effects of noise. In order to mitigate these impacts, this paper presents a novel method that combines a deterministic localization approach with a spatial feature learning process. Here, we (i) obtain a composite feature vector derived from analysing the mutual information between different spatial cues and (ii) estimate the optimum feature combination that minimizes the angular localization error in three dimensional space. The performance of the proposed mutual information based feature learning approach is evaluated for a range of speech inputs and noise conditions. We also demonstrate that the proposed approach improves the localization accuracy and its robustness, with respect to traditional localization algorithms, especially in the relatively low signal-to-noise ratio localization scenarios.

In the absence of a commercial High Dynamic Range (HDR) distribution pipeline, two-layer backward-compatible HDR video coding is a viable solution for the imminent transition from Low Dynamic Range (LDR) to HDR content transmission. However, the performance of a two-layer coding solution is governed by the extension layer coding performance. In this paper, we propose an improved two-layer backward-compatible HDR video coding solution based on an adaptive residual mapping for the extension layer, keeping in view the performance of High Efficiency Video Coding (HEVC) being used to code this information. The proposed solution outperforms the reference method achieving averaged PU-PSNR improvements of up to 5.05 dB. The proposed method also shows potential of achieving the same HDR quality as the single layer coding solution with a minimum bitrate overhead and acceptable LDR quality in the base layer.

The rising demand for media consumption via mobile devices and the emergence of complex video coding algorithms present an additional challenge for the energy management algorithms in resource-constrained consumer electronic devices. Thus, generating energy-optimized video bit streams will positively contribute towards overcoming this challenge. This paper introduces a novel energy model for intra-frame decoding of HEVC encoded video that predicts the decoding energy of a coding unit. Thereafter, a novel energy-rate-distortion optimized coding mode selection algorithm is proposed to generate energy-efficient bit streams using the proposed energy model within the encoder. The proposed energy model is shown to predict the decoding energy of a coding unit with an average error less than 2%. Moreover, the proposed coding mode selection algorithm achieves an average 10.8% reduction in the energy consumed at the decoder with a −0.25 dB impact to the Bj⊘ntegaard Delta-Peak Signal-to-Noise Ratio.

Binaural localization of speech sources in 3-D, using head-related transfer functions (HRTFs), always suffers elevation ambiguity due to the limited high frequency spectral information available at the receivers. This paper presents a method that overcomes this limitation by exploiting the interaural phase and magnitude features present in the HRTF. We (i) introduce a new feature vector that combines these two sets of features in a non-linear fashion, and (ii) propose a mechanism to extract this feature vector free from distortion by the speech spectra. The performance of the proposed method is evaluated and compared with a correlation-based HRTF database matching approach and a two-step localization technique for multiple source positions, HRTFs (individuals) and speech inputs. The results suggest that up to 20% improvement in localization performance can be achieved for moderate signal-to-noise ratios.

Social interaction of groups of users, amongst themselves and with the media content itself, is increasingly becoming popular due to the advancements in the Internet access technologies. However, multimedia resource provisioning for dispersed user groups poses a challenge and demands innovative technologies. This paper proposes a novel approach based on Particle Swarm Optimization (PSO) to optimally allocate computational and networking resources to a group of interactive users, such that the group Quality-of-Service (QoS) is maximized. We evaluate the performance of the proposed improved PSO method with respect to the state-of-the-art greedy resource allocation mechanisms and related PSO approaches. The ability to find a feasible solution (i.e., the serving probability) and the accuracy of such solutions are compared for different network topologies. The proposed method demonstrates reduced computational complexity, an up to 40% increase in the serving probability compared to the greedy methods, and up to 60 times faster convergence compared to the basic PSO approach. Overall, the comparable QoS level to the optimal solution suggests that the proposed solution efficiently allocates the resources available in the network.

High Dynamic Range (HDR) is a key technology envisioned to provide the end-users a sense of “being there”. With the advent of HDR displays in the near future, it is not clear if the existing content distribution environment will be capable of supporting HDR. Although it is generally argued that single-layer HDR coding solutions are superior to two-layer backward-compatible techniques, these coding strategies have not been fully investigated in terms of their rate distortion trade-off. In this paper, three main HDR coding techniques are implemented to analyze their performance and to evaluate the potential of HDR content distribution, while minimizing the impact on existing infrastructure, i.e., in terms of bitrate and HDR quality trade-off. The results reveal that backward-compatible solutions can outperform the single-layer approach achieving PSNR_DE improvements of up to 3.05dB on average. The HDR-VDP-2 quality indexes show similar improvements further confirming the superior performance of the backward-compatible solutions.

The Internet of Things (IoT) envisions billions of sensors deployed around us and connected to the Internet, where the mobile crowd sensing technologies are widely used to collect data in different contexts of the IoT paradigm. Due to the popularity of Big Data technologies, processing and storing large volumes of data have become easier than ever. However, large-scale data management tasks still require significant amounts of resources that can be expensive regardless of whether they are purchased or rented (e.g., pay-as-you-go infrastructure). Further, not everyone is interested in such large-scale data collection and analysis. More importantly, not everyone has the financial and computational resources to deal with such large volumes of data. Therefore, a timely need exists for a cloud-integrated mobile crowd sensing platform that is capable of capturing sensors data, on-demand, based on conditions enforced by the data consumers. In this paper, we propose a context-aware, specifically, location and activity-aware mobile sensing platform called context-aware mobile sensor data engine (C-MOSDEN) for the IoT domain. We evaluated the proposed platform using three real-world scenarios that highlight the importance of selective sensing. The computational effectiveness and efficiency of the proposed platform are investigated and are used to highlight the advantages of context-aware selective sensing.

Mitigating the effects of reverberation is a significant challenge for real-world spatial soundfield reproduction, but the necessity of a large number of reproduction channels increases the complexity and presents several challenges to existing listening room compensation techniques. In this paper, we present an adaptive room compensation method to overcome the effects of reverberation within a region, using a model description of the reverberant soundfield. We propose the reverberant channel estimation and compensation be carried out in a single step using completely decoupled adaptive filters; thus, reducing the complexity of the overall process. We compare the soundfield reproduction performance with existing adaptive and nonadaptive room compensation methods through several simulation examples. The performance of the proposed method is comparable to existing techniques, and achieves a normalized wideband region reproduction error of 1% at a signal-to-noise ratio of 50 dB, within a 1 m radius region of interest using 60 loudspeakers and 55 microphones at frequencies below 1 kHz. Robust behavior of the room compensator is demonstrated down to direct-to-reverberant-path power ratios of -5 dB. Overall, the results suggest that the proposed method can diagonalize the room compensation system, leading to a more robust and parallel implementation for spatial soundfield reproduction.

The equalization of reverberation effects is essential for spatial soundfield reproduction, but estimation of the reverberant channel presents several challenges to existing equalization techniques. This paper presents a method of active acoustic echo cancellation (AEC) for soundfield reproduction applications, using a modal description of the reverberant soundfield. We describe how individual modes of the measured soundfield can be equalized adaptively, thus reducing the complexity of the channel estimation process. AEC and reproduction performance is compared with existing adaptive and non-adaptive equalization techniques through simulation examples. Equalization performance is comparable to existing methods, achieving a normalized region reproduction error of 1% and echo return loss enhancement of 15 - 30 dB at 50 dB SNR. The results suggest that the proposed model can be used to obtain a parallel implementation of a room equalizer for active AEC.

Sensor arrays mounted on complex-shaped rigid bodies are a common feature in many practical broadband direction of arrival (DOA) estimation applications. The scattering and reflections caused by these rigid bodies introduce complexity and diversity in the frequency domain of the channel transfer function, which presents several challenges to existing broadband DOA estimators. This paper presents a novel high resolution broadband DOA estimation technique based on signal subspace decomposition. We describe how broadband signals can be decomposed into narrow subband components, and combined such that the frequency domain diversity is retained. The DOA estimation performance is compared with existing techniques using a uniform circular array and a sensor array on a hypothetical rigid body. An improvement in closely spaced source resolution of up to 6 dB is observed for the sensor array on the hypothetical rigid body, in comparison to the uniform circular array. The results suggest that frequency domain diversity, introduced by complex-shaped rigid bodies, can provide higher resolution and clearer separation of closely spaced broadband sound sources.