Panoramic video is considered to be an attractive video format since it provides the viewers with an immersive experience. However, only the viewers’ focused region of a panoramic video, viewport, is shown on the screen. Numerous applications may benefit from the future viewport prediction, such as the adaptive viewport delivery. Two aspects may be considered for predicting the viewports. Firstly, the viewport trajectory within a time interval is highly correlated, hence the historical viewport trajectory should be utilized. Secondly, the panoramic video content within the viewport highly correlates with human visual attention. In this paper, we propose the Emotional Attention map aided panoramic Viewport Estimation (EAVE) model for predicting the future viewport trajectory. In this model, the historical viewport trajectory and the emotional attention map are jointly exploited by a Long Short-Term Memory (LSTM) network. Our experiments show that the proposed method outperforms several baseline methods in terms of prediction accuracy on a panoramic dataset containing viewing trajectory data from 50 viewers for watching 75 panoramic videos.

360-degree video provides the users with immersive experience of the designated scene. However, the challenge is that delivery of the full high-quality 360-degree videos may exhaust the bandwidth leading to degraded user experience. Moreover, unreasonable number of tiles results in uneven distribution of resources during transmitting. In this paper, we propose a Two-Stage deep reinforcement learning system for 360-degree video streaming, named TS360, which determines the division of tiles and the quality allocation in two successive stages. In the tile division stage, the tiles are divided according to the viewing region weights so that the bandwidth can be utilized more efficiently in the quality allocation stage. On the other hand, we employ the Asynchronous Advantage Actor-Critic (A3C) with the aid of ResNet for optimizing the Quality of Experience (QoE) and the bandwidth efficiency. Simulation results compared with the state-of-the-art benchmarkers indicate that TS360 increases the bandwidth efficiency by 56% while improving the user QoE by 10% on average.

Panoramic video is considered to be an attractive video format, since it provides the viewers with an immersive experience, such as virtual reality (VR) gaming. However, the viewers only focus on part of panoramic video, which is referred to as viewport. Hence, the resources consumed for distributing the remaining part of the panoramic video are wasted. It is intuitive to only deliver the video data within this viewport for reducing the distribution cost. Empirically, viewports within a time interval are highly correlated, hence the historical trajectory may be used for predicting the future viewports. On the other hand, a viewer tends to sustain attention on a specific object in a panoramic video. Motivated by these findings, we propose a deep learning-based viewport Prediction scheme, namely HOP, where the Historical viewport trajectory of viewers and Object tracking are jointly exploited by the long short-term memory (LSTM) networks. Additionally, our solution is capable of predicting multiple future viewports, while a single viewport prediction was supported by the state-of-the-art contributions. Simulation results show that our proposed HOP scheme outperforms the benchmarkers by up to 33.5% in terms of the prediction error.

In this paper buffer-aided Opportunistic Routing (OR) was designed with the aid of the novel concept of linear Transmission Activation Probability Space (TAPS) partitioning invoked for relay-assisted networks, which combines the benefits of both OR [1] and of buffer-aided transmissions [2]. More specifically, a packet may be transmitted from the Source Node (SN) to the Destination Node (DN) either directly or indirectly via one of theMRelay Nodes (RNs), depending on the instantaneous channel qualities. The above-mentioned linear multi-dimensional TAPS partitioning concept is proposed for partitioning the transmission space into (2M+1) transmission regions plus an outage region, while ensuring that the number of input packets is equal to the number of output packets at each RN's buffer. The benefit of having a buffer and tolerating the associated delay is that the best channel is activated for transmission based on our linear TAPS partitioning method.

A buffer-aided two hop link is studied, where the RN is capable of temporarily storing the received packets. We commerce by defining the concept of a two-dimensional Channel Probability Space (CPS) based on the source-relay and relay-destination channel. Specifically, a non-linear CPS division method is proposed, which partitions the CPS into several regions representing the quality of the specific channels plus an outage region. Then the best channel is activated for the sake of minimizing the system's energy dissipation. Finally, the proposed buffer-aided transmission scheme relying on our non-linear CPS division regime is investigated and the results show that at given average end-to-end energy dissipation, the outage probability was reduced by 33.5% compared to the benchmark scheme.

A joint source-channel coding has attracted substantial attention with the aim of further exploiting the residual correlation residing in the encoded video signals for the sake of improving the reconstructed video quality. In our previous paper, a first-order Markov process model was utilized as an error concealment tool for exploiting the intra-frame correlation residing in the Wyner-Ziv (WZ) frame in the context of pixel-domain distributed video coding. In this contribution, we exploit the interview correlation with the aid of an interview motion search in distributed multi-view video coding (DMVC). Initially, we rely on the system architecture of WZ coding invoked for multiview video. Then, we construct a novel mesh-structured pixel-correlation model from the inter-view motion vectors and derive its decoding rules for joint source-channel decoding. Finally, we benchmark the attainable system performance against the existing pixel-domain WZ coding based DMVC scheme, where the classic turbo codec is employed. Our simulation results show that substantial bitrate reductions are achieved by employing the proposed motion-aware mesh-structured correlation modelling technique in a DMVC scheme.

Layered video coding is capable of progressively refining the reconstructed video quality with the aid of multiple layers of unequal importance. When the base layer (BL) is corrupted or lost due to channel impairments, the enhancement layers (ELs) must be discarded by the video decoder, regardless whether they are perfectly decoded or not, which implies that the transmission power assigned to the ELs is wasted. To circumvent this problem, we proposed a bit-level inter-layer forward error correction (IL-FEC) scheme for layered video transmission in our previous work, which implanted the systematic bits of the BL into the systematic bits of the ELs using exclusive-OR operations (XOR). This allowed the receiver to exploit the implanted bits of the ELs for assisting the BL's decoding and hence improved the overall system performance of our IL-FEC aided layered video scheme. In this treatise, we find the specific FEC coding rates in a real-time on-line fashion for the sake optimizing the overall system performance. The proposed procedure is widely applicable to diverse wireless transceivers and FEC codecs. Our simulation results show that the proposed optimized IL-FEC system outperforms the traditional optimal UEP by about 1.9 dB of Eb/N0at a peak signal-to-noise ratio (PSNR) of 38 dB. Viewing the improvements in terms of the video quality, 3.3 dB of PSNR improvement is attained at an Eb/N0of 10 dB, when employing a recursive systematic convolutional (RSC) code.

In layered video streaming, the enhancement layers (ELs)must be discarded by the video decoder, when the base layer (BL) is corrupted or lost due to channel impairments. This implies that the transmit power assigned to the ELs is wasted, when the BL is corrupted. To combat this effect, in this treatise we investigate the inter-layer turbo (IL-turbo) code, where the systematic bits of the BL are implanted into the systematic bits of the ELs at the transmitter. At the receiver, when the BL cannot be successfully decoded, the information of the ELs may be utilized by the IL-turbo decoder for the sake of assisting in decoding the BL. Moreover, for providing further insights into the IL technique the benefits of the IL-turbo scheme are analyzed using extrinsic information transfer (EXIT) charts in the scenario of unequal error protection (UEP) coded layered video transmission. Finally, our data partitioning based experiments show that the proposed scheme outperforms the traditional turbo code based UEP scheme by about an Eb=N0 of 1.1 dB at a peak signal-to-noise ratio (PSNR) of 36 dB or 3 dB of PSNR at an Eb=N0 of -5.5 dB at the cost of a complexity increase of 13%.

In this treatise, we investigate an inter-layer (IL) decoded self-concatenated coded system conceived for scalable video coding (SVC). In SVC, when the base layer (BL) is corrupted due to channel-induced decoding errors, the enhancement layers (ELs) are discarded by the SVC decoder. The proposed IL coding technique implants the systematic information of the BL into the systematic bits of the ELs using exclusive-OR (XOR) operations. The IL coding is combined with a self-concatenated code, which is a near-capacity iterative detection aided channel codec using a single encoder and a single decoder. At the receiver, the IL decoding technique is activated for correcting the errors residing in the error-infested BL. In our experiments, the proposed system is combined with a number of the traditional unequal error protection (UEP) arrangements for benchmarking the system performance. Our simulation results demonstrate that the proposed technique is capable of further improving the system performance of the traditional UEP assisted system by an Eb/N0of about 3 dB at a peak signal-to-noise ratio (PSNR) of 36 dB. When viewed from a different perspective, the attainable gain can be quantified as a PSNR improvement of 2.9 dB at Eb/N0= -5 dB, while imposing a marginal complexity increase of 1.5%.

Iterative source-channel decoding (ISCD) exploits the residual redundancy of the source by iteratively exchanging extrinsic information with the channel codecs to achieve robust transmission. In this treatise we propose a video codec suitable for lossless video compression and IS CD. At the encoder the spatio-temporal redundancy is partially removed at a low complexity by simply evaluating the frame difference (FD), which is then variable-length encoded (VLC). At the receiver we invoke a three-stage concatenated IS CD scheme for exploiting the residual redundancy of the FD. A Markov Random Field (MRF) model based soft-in-soft-out (SISO) module is proposed for exploiting the spatial correlations amongst the adjacent video pixels, using intra-frame coding. More explicitly, the VLC SISO decoder operates by exchanging soft information with both the MRF module and the soft channel decoder. The convergence of the three-stage iterative decoding process is examined using 3D extrinsic information transfer (EXIT) charts. Finally, we show that our system exhibits a substantial Eb/NOimprovement of about 5.2 dB compared to the corresponding benchmarker schemes.

Since joint source‐channel decoding is capable of exploiting the residual redundancy in the source signals for improving the attainable error resilience, it has attracted substantial attention. In this treatise, the authors study iterative source‐channel decoding (ISCD) aided video communications, where the video signal redundancy is modelled by a first‐order Markov process. Firstly, the authors derive reduced‐complexity formulas for the first‐order Markov modelling (FOMM) aided source decoding. Then they propose a bit‐based iterative horizontal–vertical scanline model (IHVSM) aided source decoding algorithm, where a horizontal and a vertical source decoder are employed for exchanging their extrinsic information using the iterative decoding philosophy. The iterative IHVSM aided decoder is then employed in a forward error correction (FEC) encoded uncompressed video transmission scenario, where the IHVSM and the FEC decoder exchange softbit‐information for performing turbo‐like ISCD for the sake of improving the reconstructed video quality. Finally, the authors benchmark the attainable system performance against a near‐lossless H.264/AVC video communication system and the existing FOMM‐based softbit source decoding scheme. The authors simulation results show that E b / N 0 improvements in excess of 2.8 dB are attainable by the proposed technique in uncompressed video applications.

Layered video coding creates multiple layers of unequal importance that enables us to progressively refine the reconstructed video quality. When the base layer (BL) is corrupted or lost during transmission, the enhancement layers (ELs) must be dropped, regardless of whether they are perfectly decoded or not, which implies that the transmission power assigned to the ELs is wasted. In this treatise, we propose an inter-layer forward error correction (FEC) coded video transmission scheme for mobile TV. At the transmitter, the proposed interlayer (IL) coding technique implants the systematic information of the BL into the ELs by using exclusive-OR operations. At the receiver, the implanted bits of the ELs may be utilized for assisting in decoding the BL. Furthermore, the data partition mode of H.264 video coding is utilized as the source encoder, where the type B and type C partitions will assist in protecting the type A partition. The IL coded bitstream will then be modulated and transmitted over a multifunctional multiple input multiple output (MF-MIMO) scheme for the sake of improving the system's performance in mobile environments. The proposed system may be readily combined with the traditional unequal error protection (UEP) technique, where extrinsic mutual information (MI) measurements are used for characterizing the performance of our proposed technique. Finally, our simulation results show that the proposed system model outperforms the traditional UEP aided system by about 2.5 dB of Eb/N0or 3.4 dB of peak signal-to-noise ratio (PSNR) at the cost of a 21% complexity increase, when employing a recursive systematic convolutional code. Furthermore, unlike the traditional UEP strategies, where typically stronger FEC-protection is assigned to the more important layer, employing our proposed IL coding technique requires weaker FEC to the more important layer. For example, the system relying on channel coding rates of 0.85, 0.44, and 0.44 for the type A, type B, and type C H.264 video partitions, respectively, achieves the best system performance when employing a recursive systematic convolutional (RSC) code.

Layered video coding creates multiple layers of unequal importance, where the enhancement layers will be affected when the base layer is corrupted. In this treatise, a novel inter-layer FEC scheme is investigated, where the information of the base layer1is incorporated into the systematic bits of the enhancement layers with the aid of an exclusive-OR operation. When the base layer can be recovered independently, the soft information of the enhancement layers can be deduced by flipping the sign of the check information received. In this case, the protection strength of the enhancement layer is unaffected and no extra protection bits are required. Otherwise, the inter-layer FEC decoding philosophy related to the base layer and the enhancement layer will be activated to assist in decoding the base layer, where the protection of the enhancement layer is additionally exploited to protect the base layer. Data partitioning based experiments show that our proposed scheme outperforms the traditional unequal error protection FEC aided transmission system by about 1.8 dB of channel SNR or 7.7 dB of PSNR at an acceptable complexity.

Since multiview video communications facilitate the selection of several camera views of a given scene, it may be deemed to be a promising technique for mobile television and camera-phone networks. Similar to conventional single-view video, multiview video also suffers from packet loss events imposed by network congestion. Multiple description coding (MDC) constitutes an attractive candidate solution for mitigating the packet loss events in conventional video communications. Hence it may also be deemed to be an attractive solution for multiview video. In this paper, we propose a novel tree-structured MDC (T-MDC) scheme, which can be readily implemented with the aid of arbitrary video codecs, including multiview codecs. Moreover, the proposed philosophy enables the flexible creation of a variable number of unequal importance descriptions. Finally, we will conceive a joint multiple description coding and multiview video coding (MVC) scheme. We compare two specific tree- structured MDC schemes, namely a so-called height- one complete tree-structured MDC (HOCT-MDC) and a binary tree-structured MDC (BT-MDC) scheme, which we benchmarked against simulcasting aided single description coding in the context of both single and multiview video.