Motion-compensated prediction is used in video coding standards like High Efficiency Video Coding (HEVC) as one key element of data compression. Commonly, a purely translational motion model is employed. In order to also cover non-translational motion types like rotation or scaling (zoom), e. g. contained in aerial video sequences such as captured from unmanned aerial vehicles (UAV), an affine motion model can be applied. In this work, a model for affine motion-compensated prediction in video coding is derived. Using the rate-distortion theory and the displacement estimation error caused by inaccurate affine motion parameter estimation, the minimum required bit rate for encoding the prediction error is determined. In this model, the affine transformation parameters are assumed to be affected by statistically independent estimation errors, which all follow a zero-mean Gaussian distributed probability density function (pdf). The joint pdf of the estimation errors is derived and transformed into the pdfof the location-dependent displacement estimation error in the image. The latter is related to the minimum required bit rate for encoding the prediction error. Similar to the derivations of the fully affine motion model, a four-parameter simplified affine model is investigated. Both models are of particular interest since they are considered for the upcoming video coding standard Versatile Video Coding (VVC) succeeding HEVC. Both models provide valuable information about the minimum bit rate for encoding the prediction error as a function of affine estimation accuracies.

In this work, we derive the rate-distortion function for video coding using affine global motion compensation. We model the displacement estimation error during motion estimation and obtain the bit rate after applying the rate-distortion theory. We assume that the displacement estimation error is caused by a perturbed affine transformation. The 6 affine transformation parameters are assumed statistically independent, with each of them having a zero-mean Gaussian distributed estimation error. Based on that, the joint p.d.f. of the displacement estimation errors is derived and related to the prediction error. Using the ratedistortion theory, we calculate the bit rate in dependence of the perturbation of the affine transformation parameters. Comparing with a translational motion model in video coding standards like HEVC, we determine accuracy boundaries for the affine transformation, with which a gain can be achieved.

This paper aims at simplified high dynamic range (HDR) image generation with non-modified, conventional camera sensors. One typical HDR approach is exposure bracketing, e.g. with varying shutter speeds. It requires to capture the same scene multiple times at different exposure times. These pictures are then merged into a single HDR picture which typically is converted back to an 8-bit image by using tone-mapping. Existing works on HDR imaging focus on image merging and tone mapping whereas we aim at simplified image acquisition. The proposed algorithm can be used in consumer-level cameras without hardware modifications at sensor level. Based on intermediate samplings of each sensor element during the total (pre-defined) exposure time, we extrapolate the luminance of sensor elements which are saturated after the total exposure time. Compared to existing HDR approaches which typically require three different images with carefully determined exposure times, we only take one image at the longest exposure time. The shortened total time between start and end of image acquisition can reduce ghosting artifacts. The experimental evaluation demonstrates the effectiveness of the algorithm.

In this work, we derive the rate-distortion function for video coding using the simplified affine, 4-parameter motion compensation model as it is used in the Joint Exploration Model (JEM) by the Joint Video Exploration Team (JVET) on Future Video coding. We model the displacement estimation error during motion estimation and obtain the bit rate by applying the rate-distortion theory. We assume that the displacement estimation error is caused by perturbed parameters of the simplified affine model. These transformation parameters are assumed statistically independent, with each of them having a zero-mean Gaussian distributed estimation error. The joint probability density function (p.d.f.) of the displacement estimation errors is derived and related to the prediction error. We calculate the bit rate as a function of the accuracy of the parameter estimation for the simplified affine motion model. Finally, we compare our results with a translational motion model as used in video coding standards like HEVC as well as with a full affine motion model with 6 degrees of freedom. For aerial sequences containing distinct affine motion, the minimum required bit rate to encode the prediction error can be significantly reduced from 2.5bit/sampleto 0.02 bit/sample for a reasonable operating point and a block size of 64×64 pel2.

For the generation of overview panoramic images from aerial surveillance videos, registered video frames are stitched together. Assuming a planar landscape, feature points can be detected and used to estimate a homography. However, if the features are affected by radial distortion, their mapping depends on their position within the frame and the resulting homography becomes inaccurate. As a result, the length of aerial panorama images is typically restricted to several hundred frames. To overcome this issue, we derive a model for the joint estimation of several homographies and one constant radial distortion. Due to the computational complexity of the solution, we propose a fast, iterative algorithm. Based on geometrical constraints, we regularize the projection of a jointly estimated picture group. We present panorama images from uncalibrated aerial videos with more than 1500 frames.

For low bit rate scenarios (video conferencing, aerial surveillance), conventional video coding is unable to meet the small bit rate and high quality requirements. In contrast to that Region of Interest (ROI) coding provides an efficient compression by improving the quality of ROIs at the expense of non-ROIs. We also transmit ROI only, but reconstruct non-ROI from already transmitted content by means of global motion compensation in order to provide a high quality for the full frame. Previous ROI coding systems modified the video codec to control the coding of individual blocks. We propose a codec and ROI detector independent pre- and postprocessing framework instead. This enables the usage of off-the-shelf hard-/software and an easy adaption to the latest video coding technology. Maintaining the performance of subsequent computer vision tasks, we reduce the bit rate by 90-95 % to less than 1 Mbit/s using HEVC for full HDTV videos.

Region of Interest (ROI) coding is a common method for data reduction in scenarios where bandwidth is crucial like in aerial video surveillance from Unmanned Aerial Vehicles (UAVs). In order to save bits, non-ROI areas are typically reduced in quality or not transmitted at all and thus, an accurate ROI classification is mandatory. Moving objects (MOs) are often considered as ROIs and consequently have to be accurately detected onboard. However, common detection approaches either rely on computationally demanding processing which is not available at small UAVs with only limited energy, are model based or cannot provide a sufficient detection precision. While not detected MOs lead to a degraded representation at the decoder, erroneously detected MOs lead to an unnecessary high bit rate. We tackle all these issues utilizing an efficient object proposal computation. Based on a dual-threshold strategy applied to image differences, we propose a linear prediction-supported block matcher. Compared to a simple thresholding approach, it shows superior performance and is robust to threshold tuning. By integrating superpixels into the framework, we further recover the complete shape of the MOs. Finally, an efficient tracking-by-detection system is employed to produce accurate detections from the proposals, thereby recovering missed MOs and denying wrong proposals, making the coding more efficient. We achieve an improved detection precision of up to 76 % compared to a simple difference image-based approach. By using a general ROI coding framework we reduce the bit rate of our test set by 70 % compared to common HEVC.

Low bit rate coding systems for the transmission of high quality aerial surveillance videos captured from UAVs are of high interest. One way to achieve high quality low bit rate video is to assume a planar surface of the earth, which is valid for sequences captured at high flight altitudes. Those systems only transmit the area of the current frame not contained in the previous frames (New Area) and reconstruct the already known areas by means of Global Motion Compensation (GMC) at decoder side. Although the bit rate can be reduced significantly compared to standardized video coders, no reconstruction of stereo video is possible at the decoder since each image pixel is transmitted only once and thus no motion parallax of objects can be observed in the reconstructed video. In this paper we present a coding system for stereo video reconstruction at very low bit rates. On-board the UAV we employ the camera path estimated from the image data to create a second view of a virtual camera. We derive convenient baseline distances and demonstrate the resulting perceptively good stereo impression for different test sequences. Similar to the coding concept introduced above we transmit a second New Area 2 in addition to the New Area already introduced. By doubling the bit rate to about 2Mbit/sfor a reasonable video quality of more than 38 dB, still saving more than 85% BD-rate compared to common HEVC coding, we are able to reconstruct a full HDTV (30 fps) stereo video at the decoder.

A major constraint for mobile surveillance systems (e.g. UAV-mounted) is the limited available bandwidth. Thus, ROI-based video coding, where only important image parts are transmitted with high data rate, seems to be a suitable solution for these applications. The challenging part in these systems is to reliably detect the regions of interest (ROI) and to select the corresponding macroblocks to be encoded in high quality. Recently, difference image-based moving object detectors have been used for this purpose. But as these pixel-based approaches fail for homogeneous parts of moving objects, which leads to artifacts in the decoded video stream, this paper proposes the use of a superpixel-based macroblock selection. It is shown that the proposed approach significantly reduces artifacts by raising the moving objects detection rate from 7 % to over 96 % while increasing the data rate by only 20% to 1.5-4Mbit/s (full HD resolution, 30 Hz) still meeting the given constraints. Moreover, other simple approaches (dilation) as well as state-of-the-art GraphCut-based approaches are outperformed.

One main element of modern hybrid video coders consists of motion compensated prediction. It employs spatial or temporal neighborhood to predict the current sample or block of samples, respectively. The quality of motion compensated prediction largely depends on the similarity of the reference picture block used for prediction and the current picture block. In case of varying blur in the scene, e.g. caused by accelerated motion between the camera and objects in the focal plane, the picture prediction is degraded. Since motion blur is a common characteristic in several application scenarios like action and sport movies we suggest the in-loop compensation of motion blur in hybrid video coding. Former approaches applied motion blur compensation in single layer coding with the drawback of needing additional signaling. In contrast to that we employ a scalable video coding framework. Thus, we can derive strength as well as the direction of motion of any block for the high quality enhancement layer by base-layer information. Hence, there is no additional signaling necessary neither for predefined filters nor for current filter coefficients. We implemented our approach in a scalable extension of the High Efficiency Video Coding (HEVC) reference software HM 8.1 and are able to provide up to 1% BD-Rate gain in the enhancement layer compared to the reference at the same PSNR-quality for JCT-VC test sequences and up to 2.5% for self-recorded sequences containing lots of varying motion blur.

Current Moving Object Detectors in airborne Region of Interest (ROI) coding systems for police surveillance applications used on-board of UAVs are often based on Global Motion Estimation (GME) techniques. Since in these scenarios the camera is moving, simple background removal approaches cannot be applied without a Global Motion Compensation (GMC). Common GMC algorithms assume the ground to be planar, allowing the pixels of the previous frame to be motion compensated into the current frame by applying a projective transformation. The difference image between the compensated frame and the current frame emphasis regions containing possible motion. Such moving object detectors are great in terms of run-time efficiency but are known to lack in terms of accuracy - especially for unstructured regions of moving objects - as well as the robustness against noise. Superpixel segmentation was recently proposed to overcome the issue of the imprecise region cuts given by the difference image. It provides a greatly improved true positive detection rate, but unintentionally also increases the area of false positives. This paper proposes the use of a mesh-based GME and GMC to detect the moving object regions wherein a cluster filter eliminates errors in the optical flow by assuming a smooth vector field as the global motion model. In doing so we improve the coding efficiency of the fully automatic ROI coding system by more than 24% for moving object areas conserving the detection benefits of the integration of superpixel segmentation.

Current video coding standards perform well for video sequences captured by a real camera. The aperture of the camera's optical system smooths the content and attenuates higher frequencies. New application scenarios, enabled by the growing number of high bit rate internet gateways, however, make it necessary to take a closer look at the efficiency of such standards in handling artificial content. Remote desktop applications for example often include text parts. As a consequence, these content types contain sharp edges or high frequencies, which are considered less important in natural video and are therefore treated less carefully. The frequent result is an increased occurrence of artefacts or the loss of information that is actually important to the user. This paper gives an analysis of such artificially created video sequences, evaluates the performance of current coding tools for this type of content and proposes a simple, yet effective way to maintain readability of text within video material using only well considered encoder control and without the need of large additional modules.