Raw images are commonly processed by image signal processing (ISP) algorithms to standard RGB (sRGB) images in order to save the storage space and provide a suitable format for human visual system. Distortion and information loss are introduced in this procedure which degrades the performance of following tasks with these sRGB images as the input. To leverage the benefits from raw images, reversing sRGB images to raw images is expected. In this paper, a progressive physics-driven deep learning algorithm is proposed for the conversion of sRGB images to higher quality raw images by fusing physical-driven and data-driven deep learning approaches. Based on an observation that deep convolution neural networks (CNNs) are biased towards learning low-frequency functions, the proposed framework includes a high-frequency aware guidance branch to provide progressive guidance for the reconstruction branch. Experimental results indicate that the proposed algorithm improves reconstruction quality of existing data-driven approaches. It also reduces the sensitivity of existing data-driven approaches to test data in the sense that those images that are hard to be restored by the existing approaches are improved much more.

Model-based single image dehazing algorithms restore images with sharp edges and rich details at the expense of low PSNR values. Data-driven ones restore images with high PSNR values but with low contrast, and even some remaining haze. In this paper, a novel single image dehazing algorithm is introduced by integrating model-based and data-driven approaches. Both transmission map and atmospheric light are initialized by the model-based methods, and refined by deep learning based approaches which form a neural augmentation. Haze-free images are restored by using the transmission map and atmospheric light. Experimental results indicate that the proposed algorithm can remove haze well from real-world and synthetic hazy images.

Model-based single image dehazing algorithms restore haze-free images with sharp edges and rich details for real-world hazy images at the expense of low PSNR and SSIM values for synthetic hazy images. Data-driven ones restore haze-free images with high PSNR and SSIM values for synthetic hazy images but with low contrast, and even some remaining haze for real-world hazy images. In this paper, a novel single image dehazing algorithm is introduced by combining model-based and data-driven approaches. Both transmission map and atmospheric light are first estimated by the model-based methods, and then refined by dual-scale generative adversarial networks (GANs) based approaches. The resultant algorithm forms a neural augmentation which converges very fast while the corresponding data-driven approach might not converge. Haze-free images are restored by using the estimated transmission map and atmospheric light as well as the Koschmieder's law. Experimental results indicate that the proposed algorithm can remove haze well from real-world and synthetic hazy images.

Model driven single image dehazing was widely studied due to its broad applications. It is challenging to prevent noise from being amplified in sky region for the model driven dehazing algorithms. In this paper, a new multi-scale hazy image model is first built up by using the Laplacian pyramid of hazy image and the Gaussian pyramid of transmission map. A novel multi-scale dehazing algorithm is then proposed on top of the model to address the problem. Different haze removal and noise reduction approaches are applied to restore the scene radiance at different levels of the pyramid. The resultant pyramid is collapsed to restore a haze-free image. Experiment results demonstrate that the proposed algorithm outperform state of the art dehazing algorithms and the noise is indeed prevented from being amplified in the sky region.

Model-based single image dehazing was widely studied due to its extensive applications. Ambiguity between object radiance and haze and noise amplification in sky regions are two inherent problems of model-based single image dehazing. In this paper, a dark direct attenuation prior (DDAP) is proposed to address the former problem. A novel haze line averaging is proposed to reduce the morphological artifacts caused by the DDAP which enables a weighted guided image filter with a smaller radius to further reduce the morphological artifacts while preserve the fine structure in the image. A multi-scale dehazing algorithm is then proposed to address the latter problem by adopting Laplacian and Gaussian pyramids to decompose the hazy image into different levels and applying different haze removal and noise reduction approaches to restore the scene radiance at the different levels. The resultant pyramid is collapsed to restore a haze-free image. Experiment results demonstrate that the proposed algorithm outperforms state-of-the-art dehazing algorithms.

Spatially varying exposures via rolling shutter is an efficient way to capture differently exposed images for high dynamic range (HDR) scenes. Neither camera movement nor moving objects is an issue for such a captured method. However, a possible issue is that the resolution of captured images is reduced. In this paper, we introduce a novel cross image cubic interpolator for the spatially varying exposures via the rolling shutter. Both intra correlation among pixels with the same exposure and inter correlation among pixels with the different exposures are utilized by the proposed interpolator. Experimental results show that quality of upsampled images is significantly improved.

Urban noise pollution attracts global concern since it seriously affects and harms human health and activities. Noise-induced annoyance is one of the salient noise effects on human perception. As an efficient noise reduction tools, noise masking scheme is employed to reduce noise-induced annoyance. Both music and water sounds are commonly used sound maskers in noise masking where music sound masking is seldom discussed for urban noise. In this study, we conduct a subjective listening test to evaluate the masking capability of music and water sounds for Singapore urban noise. The results show that masking is a good solution in mid-to-high annoyance scenario for traffic and construction noise. And music sound presents similar performance in reducing the annoyance to that of the water sounds.

Convolutional neural network (ConvNet) is a class of deep feed-forward neural network which exploits the strong spatially local correlation in natural images. It achieves successful performance in visual analyzing area. Recently, ConvNet has been employed in acoustic processing area and been proved to be able to learn the spectro-temporal pattern of sound and differential them for the classification purpose. In this manuscript, the time-frequency resolution of the input sound is studied for their efficiency in the classification accuracy when ConvNet is adopted. Simulation results shows that the data augment solution, which is called multi-width frequency-delta, presents little contribution for the performance improvement when the network is carefully designed. In addition, a suitable temporal resolution in acoustic sound segmentation can achieve good classification effect.

One direct effect of ubiquitous environmental noises on humans is annoyance. It is caused by a complex mix of noise exposure quantities, non-acoustical factors and potential psycho-acoustical factors. which made it challenging to establish an objective annoyance prediction model. This paper presents our study to use state-of-the-art machine learning algorithms to develop an annoyance prediction model. The objective of this study is threefold: (a) to identify relevant audio features with the most predictive power to annoyance, (b) to build a prediction model with high-accuracy and (c) to quantify each feature's influence on annoyance. Our modeling procedure shows that with the power of machine learning, such an annoyance prediction model can be successfully built with satisfying performances on all our objectives.

This paper investigates the problem of noise-induced annoyance level evaluation, and proposes a novel annoyance measurement metric for more efficient and accurate evaluation of annoyance level of different types of noises. Results from a large-scale subjective listening test using 90 different noise clips and 96 subjects show that the proposed method can produce more consistent and reliable annoyance ratings than the widely adopted ISO method. Based on the subjective test results, we further develop an objective noise annoyance level measurement model based on the selected psycho-acoustic features extracted from the noise samples. Our evaluation results show that the objective model produces satisfactory prediction accuracy on noise annoyance level.

A memory-based search circuit is introduced in this paper. In this circuit, the conventional memory structure is customized to provide equality comparison for each column of memory array, and a counting circuit is included at each column to record the degree of matches between query and reference data patterns. This customized memory circuit can be used for similarity search applications. Due to its massive parallel processing of equality comparisons and counting operations, the search time using this circuit has O(1) complexity. In addition, it uses NOR flash memory structure and Enhanced Vote Count (EVC) interlocked design to achieve low power and high speed. Energy consumption is significantly reduced, by approximately m-fold (m is the number of simultaneously compared pattern bits in EVC), while matching speed is m times faster, compared to original vote count circuit implemented on NOR flash memory structure.

k-Nearest Neighbors (kNN) algorithm is a method to find the closest points in a dataset to a query point. The result of kNN can be used for classification and regression, both of which are commonly used in data mining and machine learning. In this paper, Enhanced Vote Count (EVC) circuit, which uses hardware to compare the quantized projected values of query and training/reference vectors instead of the vectors themselves, is considered to approximate the kNN search to provide a low complexity search solution. To improve the performance of EVC with limited projection number because projection number is directly related to implementation cost of EVC circuit, the concept of weak bit is considered and only reliable binary pattern matching is evaluated. The implementation of weak bit based on EVC circuit is also described. Simulation results show that, the performance of EVC can be significantly improved with weak bit implementation under limited projection number.

Linear prediction serves as a mathematical operation to estimate the future values of a discrete‐time signal based on a linear function of previous samples. When applied to predictive coding of waveform such as speech and audio, a common issue that plagues compression performance is the non‐stationary characteristics of prediction residuals around the starting point of the random access frames. This is because dependencies between prediction residuals and the historical waveform are interrupted to satisfy the random access requirement. In such cases, the dynamic range of the prediction residuals will fluctuate dramatically in such frames, leading to substantially poor coding performance in the subsequent entropy coder. In this study, the authors developed a solution to this long‐standing issue by establishing a theoretical relationship between the energy envelope of linear prediction residuals in the random access frames and the prediction coefficients. Using the established relationship, an adaptive normalisation method is formulated as a preprocessor to the entropy coder to mitigate the poor coding performance in the random access frames. Simulation results confirm the superiority of the proposed method over existing solutions in terms of coding efficiency performance.

We propose a time-utility function (TUF)-based packet scheduling algorithm for streaming scalable media. In the proposed system, the scalable media is partitioned into data units of different quality layers, which are then prioritized and transmitted according to their TUFs that capture both their quality contributions to the decoded media and urgencies with respect to their playback schedule. For optimal streaming quality while maintaining a reasonable computational complexity, packet transmissions are scheduled using a low-complexity algorithm based on utility accrual maximization. The computational complexity can be further reduced by considering the look-ahead window and a modified utility function. Simulation results show that the proposed scheduling algorithms achieve near-optimal performance when compared with the operational rate distortion bound of the stream source at any given bandwidth budget.

A new scalable audio coding scheme is introduced in this paper. Its core idea is to create one additional scalability dimension during the encoding process for the purpose of generating a plural of scalable sub-bitstreams. Based on the multiple sub-streams, a smart truncator is designed that can truncate these sub-bitstreams with optimal rate-distortion (R-D) tradeoff. Benefited from the flexible R-D trade-off, the proposed new scheme could, within a wide bitrate range, outperform those traditional scalable coding schemes, which usually provides a fixed R-D relationship designed at a specified bitrate. To verify the performance, the proposed scheme is further implemented based on a prior art scalable audio codec. Significant quality improvement is observed from the new codec via a series of subjective listening tests.

In this paper, a packet scheduling algorithm that is based on a Time-Utility Function (TUF) is proposed. In the proposed system, the scalable media is partitioned into data units of different quality layers, which are then prioritized and transmitted according to their TUF's that capture both their quality contributions to the decoded media and urgencies with respect to their playback schedule. For optimal streaming quality and meanwhile maintaining a reasonable computational complexity, the scheduling of packet transmission is obtained from a low-complexity packet scheduling algorithm based on utility accrual maximization. Experimental results show that the proposed scheduling algorithm achieves near optimal performance when compared to the operational rate distortion bound of the stream source at the capacity of the network.

Bit-plane probability of data is useful information for many applications such as entropy coding and rate estimation, particularly for scalable coding system. Typically, the value of bit-plane probability varies according to the distribution of the data represented, and due to inter-plane correlation, it is hard to get an analytic solution of bit-plane probability for the data with generalized Gaussian distribution. In this paper, bit-plane probability is analyzed when Generalized Gaussian distribution is used to model the input data. Based on the study of bit-plane probability for Laplace distribution and the relationship between different bit-planes, an approximated bit-plane probability for Generalized Gaussian distribution is presented. This closed-form expression is of low computational cost. Furthermore, a much practical format is derived with reduced complexity for implementation in the state-of-the-art MPEG-4 scalable to lossless audio coding system. With the same computational cost, the proposed algorithm presents higher compression efficiency than MPEG-4 scalable to lossless audio coding, which considers Laplace distribution only.

For bit-plane coding of random source, we establish a relationship between the probability density function of an arbitrary non-negative distribution and the bit-plane symbol probability. Applied to Laplacian source, the relationship gives a simple, closed form of bit-plane symbol probability, which is a superset of the probability assignment rule adopted in the MPEG-4 Audio Scalable Lossless Coding (SLS) international standard. For bit-plane arithmetic coding of Laplacian source, we propose an algorithm and apply it to MPEG-4 SLS. Experimental results show that without changing the encoder/decoder complexity, the proposed algorithm consistently improved the compression ratio over SLS for all the 51 MPEG test sequences. The overall improvement in the compression ratio is 0.11%.

Scalable to lossless (SLS) audio coding is a state-of-art audio coding technique that has been adopted as MPEG scalable audio coding tool. To realize bit-plane refinement, this technique employs bit-plane arithmetic coding for lossless entropy coding, and Laplacian distribution is used to model the input data to realize high compression efficiency. In this paper, bit-plane probability is analyzed when generalized Gaussian distribution is used to model the input data. Based on the result of bit-plane probability for generalized Gaussian distribution, a low cost bit-plane arithmetic coding method is presented. This scheme is implemented in the SLS audio coding platform. With the same computational complexity, the proposed algorithm presents higher compression efficiency than SLS.

Bit-plane probability of data is a useful characteristic for many applications, such as entropy coding. Its value may vary according to the distribution of the data. In this paper, the bit-plane probability of data with generalized Gaussian distribution is analyzed. By studying the statistical characteristics of the sources and the bit-plane probability for Laplacian distribution, an approximated analytic description of bit-plane probability for generalized Gaussian distribution is presented. This approximation calculates the bit-plane probability with low computational cost. Simulation results show that, the proposed algorithm can well approximate the actual bit-plane probability. In addition, when the proposed algorithm is applied to audio bit-plane arithmetic coding, the coding efficiency is higher than that based on the Laplacian distribution assumption.

When high-quality video is delivered via a bandwidth-limited network, one solution is to reduce the spatial resolution of the bitstream before transmission. Downsizing transcoding is adopted as an efficient tool to reduce the spatial resolution of the precoded bitstream. Since the downsizing process and the coding process both apply a low-pass filtering operation on the picture, the reconstructed video may result in serious blocking artifact. In this paper, pre- and post-shift filters are considered in the transcoding process. The block boundary is shifted between the downsizing process and the coding process. This reduces the blocking effect in downsizing transcoding. Experimental results show that the proposed scheme can efficiently remove the blocking effect without introducing high computation.

When transmitting the precoded bitstream over an error-prone network, error-resilient transcoding is adopted to convert the bitstream to a resilient format for robust delivery. Intra refreshment is an efficient tool to reduce the dependency between frames and stop the channel error propagation. In the conventional scheme, the rate-distortion optimized macroblock mode decision is employed to adaptively determine the coding mode of each macroblock. However, this scheme only considers the channel error propagated from the previous frames to the current frame. As opposed to this traditional algorithm, this paper proposes a method which considers consecutive two frames in a sequence, thus taking the error propagation to the following frame into account. This enhances the overall robustness of the transcoded bitstream against the packet loss. Considering the availability of the next frame information, two cases are discussed respectively. Experimental results show that the proposed methods present quality improvement when compared with the conventional rate-distortion optimized error-resilient coding scheme under different test environments, and the PSNR improvement can reach as high as 0.9 dB.

When the high definition video is delivered via a limited bandwidth network, one solution is to reduce the spatial resolution of the bitstream. Downsizing transcoding is adopted as an efficient tool to reduce the spatial resolution of the pre-coded bitstream. Since the downsizing process and the coding process apply a low pass filter to the picture, the reconstructed video may present serious blocking artifact. In this paper, pre-and post-shift filters are considered in the transcoding process. The block boundary is shifted between the downsizing process and the coding process. This reduces the blocking effect in downsizing transcoding. Experimental results show that the proposed scheme can efficiently reduce the blocking effect without introducing high computation.

In the H.264 coding standard, an integer cosine transform (ICT) is adopted for transform coding. As the ICT is an integer approximation to the discrete cosine transform (DCT), its performance is different from that of the DCT. When the scheme of discarding high frequency coefficients is employed in the ICT domain downsizing, the output quality is worse than when done in the DCT domain. In this paper, the authors propose a scheme to enhance the output quality by converting the downsizing operation back to the DCT domain. The experimental results show that the proposed schemes present quality enhancement over the ICT domain downsizing.

In video transcoding, arbitrary resizing is always desired for end users with different display capabilities. In this paper, a two-stage resizing structure is proposed for arbitrary resizing. It gives a general solution for the realization of arbitrary resizing in the transform domain. Two constraints are given for the selection of parameter set to benefit the anti-aliasing and picture quality. Illustrative examples show that our proposed two-stage resizing structure, with the qualified parameter sets, can provide an appropriate solution for arbitrary resizing

In video transcoding, arbitrary resizing is always desired for end users with different display capabilities. In this paper, a two-stage resizing structure is proposed for arbitrary resizing. It gives a general solution for the realization of arbitrary resizing in the transform domain. Two constraints are given for the selection of parameter set to benefit the anti-aliasing and the picture quality. Illustrative examples show that our proposed two-stage resizing structure, with qualified parameter set, can provide a satisfactory solution for arbitrary resizing.

In order to transmit the precoded bit stream through bandwidth constrained network, transcoding technique has been introduced to reduce the overall bit rate generated. Downsizing transcoding is one of the commonly used methods since it can reach very low bit rate without losing any motion information. Many efforts have been devoted to this research area. With the proposal of arbitrary downsizing, fine gradual reduction of bit rate and video quality becomes feasible. However, there is little research focusing on frame size selection for arbitrary downsizing transcoding. In this letter, the realization of arbitrary downsizing transcoding is discussed. Rate estimation scheme is applied on video sequences to choose the suitable frame size for the target bit rate. An efficient method is proposed to estimate the number of bits allocated to residue in requantization process. Experimental results show that, it can provide an accurate estimation and make a reliable decision in frame size selection.

When delivering video over communication networks, it is required to transcode the precoded video content to meet the demands of a broad range of end users with different bandwidths and resource constraints. One solution to transmit video over bandwidth-constrained channels is to reduce the spatial resolution of video frames and transmit a low-resolution version of the video as a tradeoff for the bit rate. In this paper, an arbitrary downsizing algorithm is proposed. This arbitrary downsizing algorithm is processed directly in the discrete cosine transform domain. Experimental results show that the proposed method can achieve a satisfactory performance. Compared to the existing methods, this algorithm is not only applicable to an intracoding frame but also to an intercoding frame. This is one of the crucial features of this method.

Video transmission requires the delivery of video content to a broad range of end users with different bandwidth and resource constraints. One solution to transmitting video over bandwidth-constrained channels is to transmit a low-resolution version of video as a trade off for the bit-rate. In this paper, we proposed an arbitrary downsizing algorithm. This algorithm can downsize the frame to arbitrary size and process it directly in DCT domain. Experimental results show that the proposed method can achieve a comparable performance. Compared with the existing methods, this algorithm is not only applicable to intra coding frame but also inter coding frame. This is one of the crucial features of this method.