Conventional cameras capture image irradiance (RAW) on a sensor and convert it to RGB images using an image signal processor (ISP). The images can then be used for photography or visual computing tasks in a variety of applications, such as public safety surveillance and autonomous driving. One can argue that since RAW images contain all the captured information, the conversion of RAW to RGB using an ISP is not necessary for visual computing. In this paper, we propose a novel ρ-Vision framework to perform high-level semantic understanding and low-level compression using RAW images without the ISP subsystem used for decades. Considering the scarcity of available RAW image datasets, we first develop an unpaired CycleR2R network based on unsupervised CycleGAN to train modular unrolled ISP and inverse ISP (invISP) models using unpaired RAW and RGB images. We can then flexibly generate simulated RAW images (simRAW) using any existing RGB image dataset and finetune different models originally trained in the RGB domain to process real-world camera RAW images. We demonstrate object detection and image compression capabilities in RAW-domain using RAW-domain YOLOv3 and RAW image compressor (RIC) on camera snapshots. Quantitative results reveal that RAW-domain task inference provides better detection accuracy and compression efficiency compared to that in the RGB domain. Furthermore, the proposed ρ-Vision generalizes across various camera sensors and different task-specific models. An added benefit of employing the ρ-Vision is the elimination of the need for ISP, leading to potential reductions in computations and processing times.

Screen content coding (SCC) is the latest extension of the High-Efficiency Video Coding (HEVC) aiming to improve the compression efficiency of screen content video. With newly developed tools such as intra block copy (IntraBC) and palette (PLT) mode, SCC has been able to compress the desktop screens more efficiently but with significant complexity increase. In this paper, we improve the intra prediction from two aspects. Firstly, by leveraging the temporal correlation among coding units (CU), we develop a fast CU depth prediction scheme. Furthermore, adaptive search step is employed for further speed up of the time-consuming block matching in IntraBC. The overall encoding time is reduced by about 39% and 35% for the All Intra (AI) lossy and lossless encoding scenarios with negligible quality loss under the SCC common test condition.

This work attempts to quantitatively study the impacts of the different input video source format on the coding efficiency and complexity of the recent High-Efficiency Video Coding (HEVC) based screen content compression. Besides the native RGB color space, YUV color space with 4:4:4 and 4:2:0 chroma sampling are investigated. Decoded videos are all converted to the RGB space to calculate the structural similarity as the subjective distortion estimation, and perform the subjective rating to have the Mean Opinion Score (MOS). Performance is evaluated using the bit rate versus structural similarity (SSIM), and bit rate versus MOS. Experiments have demonstrated approximately 1.3% SSIM loss with 30% bit rate reduction, and 3.4% SSIM loss with 33% bit rate reduction, respectively, when using YUV 4:4:4 and YUV 4:2:0 as input sources versus native RGB input. Similar performance is recorded for MOS versus bit rate. We have also observed 18% and 31% encoding time reduction by using YUV 4:4:4 and YUV 4:2:0 rather than RGB content for popular low-delay scenario.