Multiagent reinforcement learning (MARL) is useful in many problems that require the cooperation and coordination of multiple agents. Learning optimal policies using reinforcement learning in a multiagent setting can be very difficult as the number of agents increases. Recent solutions such as value decomposition networks (VDNs), QMIX, QTRAN, and QPLEX adhere to the centralized training and decentralized execution (CTDE) scheme and perform factorization of the joint action-value functions. However, these methods still suffer from increased environmental complexity, and at times fail to converge in a stable manner. We propose a novel concept of residual Q-networks (RQNs) for MARL, which learns to transform the individual Q -value trajectories in a way that preserves the individual-global-max (IGM) criteria, but is more robust in factorizing action-value functions. The RQN acts as an auxiliary network that accelerates convergence and will become obsolete as the agents reach the training objectives. The performance of the proposed method is compared against several state-of-the-art techniques such as QPLEX, QMIX, QTRAN, and VDN, in a range of multiagent cooperative tasks. The results illustrate that the proposed method, in general, converges faster, with increased stability, and shows robust performance in a wider family of environments. The improvements in results are more prominent in environments with severe punishments for noncooperative behaviors and especially in the absence of complete state information during training time.

Increasingly, the task of detecting and recognizing the actions of a human has been delegated to some form of neural network processing camera or wearable sensor data. Due to the degree to which the camera can be affected by lighting and wearable sensors scantiness, neither one modality can capture the required data to perform the task confidently. That being the case, range sensors, like light detection and ranging (LiDAR), can complement the process to perceive the environment more robustly. Most recently, researchers have been exploring ways to apply convolutional neural networks to 3-D data. These methods typically rely on a single modality and cannot draw on information from complementing sensor streams to improve accuracy. This article proposes a framework to tackle human activity recognition by leveraging the benefits of sensor fusion and multimodal machine learning. Given both RGB and point cloud data, our method describes the activities being performed by subjects using regions with a convolutional neural network (R-CNN) and a 3-D modified Fisher vector network. Evaluated on a custom captured multimodal dataset demonstrates that the model outputs remarkably accurate human activity classification (90%). Furthermore, this framework can be used for sports analytics, understanding social behavior, surveillance, and perhaps most notably by autonomous vehicles (AVs) to data-driven decision-making policies in urban areas and indoor environments.

Increasingly complex automated driving functions, specifically those associated with free space detection (FSD), are delegated to convolutional neural networks (CNNs). If the dataset used to train the network lacks diversity, modality, or sufficient quantities, the driver policy that controls the vehicle may induce safety risks. Although most autonomous ground vehicles (AGVs) perform well in structured surroundings, the need for human intervention significantly rises when presented with unstructured niche environments. To this end, we developed an AGV for seamless indoor and outdoor navigation to collect realistic multimodal data streams. We demonstrate one application of the AGV when applied to a self-evolving FSD framework that leverages online active machine-learning (ML) paradigms and sensor data fusion. In essence, the self-evolving AGV queries image data against a reliable data stream, ultrasound, before fusing the sensor data to improve robustness. We compare the proposed framework to one of the most prominent free space segmentation methods, DeepLabV3+ [1]. DeepLabV3+ [1] is a state-of-the-art semantic segmentation model composed of a CNN and an autodecoder. In consonance with the results, the proposed framework outperforms DeepLabV3+ [1]. The performance of the proposed framework is attributed to its ability to self-learn free space. This combination of online and active ML removes the need for large datasets typically required by a CNN. Moreover, this technique provides case-specific free space classifications based on the information gathered from the scenario at hand.

Privacy-preserving (PP) applications allow users to perform online daily actions without leaking sensitive information. The PP scalar product (PPSP) is one of the critical algorithms in many private applications. The state-of-the-art PPSP schemes use either computationally intensive homomorphic (public-key) encryption techniques, such as the Paillier encryption to achieve strong security (i.e., 128 b) or random masking technique to achieve high efficiency for low security. In this article, lattice structures have been exploited to develop an efficient PP system. The proposed scheme is not only efficient in computation as compared to the state-of-the-art but also provides a high degree of security against quantum attacks. Rigorous security and privacy analyses of the proposed scheme have been provided along with a concrete set of parameters to achieve 128-b and 256-b security. Performance analysis shows that the scheme is at least five orders faster than the Paillier schemes and at least twice as faster than the existing randomization technique at 128-b security. Also the proposed scheme requires six-time fewer data compared to the Paillier and randomization-based schemes for communications.

Humans live among other humans, not in isolation. Therefore, the ability to learn and behave in multi-agent environments is essential for any autonomous system that intends to interact with people. Due to the presence of multiple simultaneous learners in a multi-agent learning environment, the Markov assumption used for single-agent environments is not tenable, necessitating the development of new Policy Learning algorithms. Recent Actor-Critic algorithms proposed for multi-agent environments, such as Multi-Agent Deep Deterministic Policy Gradients and Counterfactual Multi-Agent Policy Gradients, find a way to use the same mathematical framework as single agent environments by augmenting the Critic with extra information. However, this extra information can slow down the learning process and afflict the Critic with Curse of Dimensionality. To combat this, we propose a novel Deep Neural Network configuration called Deep Multi-Critic Network. This architecture works by taking a weighted sum over the outputs of multiple critic networks of varying complexity and size. The configuration was tested on data collected from a real-world multi-agent environment. The results illustrate that by using Deep Multi-Critic Network, less data is needed to reach the same level of performance as when not using the configuration. This suggests that as the configuration learns faster from less data, then the Critic may be able to learn Q-values faster, accelerating Actor training as well.

The increased compression ratios achieved by the High Efficiency Video Coding (HEVC) standard lead to reduced robustness of coded streams, with increased susceptibility to network errors and consequent video quality degradation. This paper proposes a method based on a two-stage approach to improve the error robustness of HEVC streaming, by reducing temporal error propagation in the case of frame loss. The prediction mismatch that occurs at the decoder after frame loss is reduced through the following two stages. First, at the encoding stage, the reference pictures are dynamically selected based on constraining conditions and Lagrangian optimization, which distributes the use of reference pictures, by reducing the number of prediction units that depend on a single reference. Second, at the streaming stage, a motion vector (MV) prioritization algorithm, based on spatial dependencies, selects an optimal subset of MVs to be transmitted, redundantly, as side information to reduce mismatched MV predictions at the decoder. The simulation results show that the proposed method significantly reduces the effect of temporal error propagation. Compared with the reference HEVC, the proposed reference picture selection method is able to improve the video quality at low-packet-loss rates (e.g., 1%) using the same bitrate, achieving quality gains up to 2.3 dB for 10% of packet loss ratio. It is shown, for instance, that the redundant MVs are able to boost the performance achieving quality gains of 3 dB when compared with the reference HEVC, at the cost using 4% increase in total bitrate.

In this paper a robust encoding scheme is proposed to improve the visual quality of HEVC decoded video when intra frames are lost along the streaming path. For this purpose, the encoding process includes frame loss simulation and subsequent error concealment, to find the most efficient method that should be used by a decoder to recover lost intra frames. In this novel scheme, each image is divided into partitions, which are associated with the error concealment method that achieves the lowest distortion. Then this information is signalled to the decoder through SEI messages in the coded stream. In order to efficiently use the signalling overhead, rate-distortion optimisation is used to achieve the best trade-off between the number of transmitted symbols and distortion of reconstructed frames. Experimental results show the effectiveness of the proposed method to enhance the quality of reconstructed intra frames under different packet loss ratios (PLR). For PLR=10%, the robust coding scheme is able to improve the average PSNR of all frames affected by errors, up to 1.50 dB and 3.44 dB in Low-Delay and Random-Access configurations respectively, at a maximum overhead cost of 0.24%.

Multiview entertainment is the next step in 3D immersive media networking owing to its improved depth perception and free-viewpoint viewing capability whereby users can observe the scene from the desired viewpoint. This paper outlines a delivery system for multiview plus depth video, combining the broadcast and broadband networks. The digital video broadcast (DVB) network is used along with adaptive peer-to-peer (P2P) distribution over the Internet to deliver high-volume multimedia to users. The DVB network has been used to deliver part of the 3D service, owing to its robustness and wide availability, as a mechanism to guarantee the minimum 3D quality of experience. The developed system brings key contributions in the P2P transport for real-time multimedia delivery, including a user preference-aware adaptation mechanism, adaptive redundant chunk scheduling for robustness, incentives to decrease the load on the content server for improved system scalability, and resynchronization capability with the DVB transmission. The introduced features are compared with those of some other well-known P2P solutions to highlight the quantitative gains. A subjective testing campaign has also been organized on the developed hybrid platform, which proves the effectiveness of user-aware adaptation over network-based adaptation on a mean opinion score scale.

Network traffic classification is the process of analyzing traffic flows and associating them to different categories of network applications. Network traffic classification represents an essential task in the whole chain of network security. Some of the most important and widely spread applications of traffic classification are the ability to classify encrypted traffic, the identification of malicious traffic flows, and the enforcement of security policies on the use of different applications. Passively monitoring a network utilizing low-cost and low-complexity wireless local area network (WLAN) devices is desirable. Mobile devices can be used or existing office desktops can be temporarily utilized when their computational load is low. This reduces the burden on existing network hardware. The aim of this paper is to investigate traffic classification techniques for wireless communications. To aid with intrusion detection, the key goal is to passively monitor and classify different traffic types over WLAN to ensure that network security policies are adhered to. The classification of encrypted WLAN data poses some unique challenges not normally encountered in wired traffic. WLAN traffic is analyzed for features that are then used as an input to six different machine learning (ML) algorithms for traffic classification. One of these algorithms (a Gaussian mixture model incorporating a universal background model) has not been applied to wired or wireless network classification before. The authors also propose a ML algorithm that makes use of the well-known vector quantization algorithm in conjunction with a decision tree-referred to as a TRee Adaptive Parallel Vector Quantiser. This algorithm has a number of advantages over the other ML algorithms tested and is suited to wireless traffic classification. An average F-score (harmonic mean of precision and recall) > 0.84 was achieved when training and testing on the same day across six distinct traffic types.

Advances in video coding and networking technologies have paved the way for the Multi-View Video (MVV) streaming. However, large amounts of data and dynamic network conditions result in frequent network congestion, which may prevent video packets from being delivered on time. As a consequence, the 3D viewing experience may be degraded significantly, unless quality-aware adaptation methods are deployed. There is no research work to discuss the MVV adaptation of decision strategy or provide a detailed analysis of a dynamic network environment. This work addresses the mentioned issues for MVV streaming over HTTP for emerging multi-view displays. In this research work, the effect of various adaptations of decision strategies are evaluated and, as a result, a new quality-aware adaptation method is designed. The proposed method is benefiting from layer based video coding in such a way that high Quality of Experience (QoE) is maintained in a cost-effective manner. The conducted experimental results on MVV streaming using the proposed strategy are showing that the perceptual 3D video quality, under adverse network conditions, is enhanced significantly as a result of the proposed quality-aware adaptation.

The improved compression efficiency achieved by the High Efficiency Video Coding (HEVC) standard has the counter-effect of decreasing error resilience in transmission over error-prone channels. To increase the error resilience of HEVC streams, this paper proposes a checkerboard reference picture selection method in order to reduce the prediction mismatch at the decoder in case of frame losses. The proposed approach not only allows to reduce the error propagation at the decoder, but also enhances the quality of reconstructed frames by selectively constraining the choice of reference pictures used for temporal prediction. The underlying approach is to increase the amount of accurate temporal information at the decoder when transmission errors occur, to improve the video quality by using an efficient combination of diverse motion fields. The proposed method compensates for the small loss of coding efficiency at frame loss rates as low as 3%. For a single frame-loss event the proposed method can achieve up to 2 dB of gain in the affected frames and an average quality gain of 0.84 dB for different error prone conditions.

With the exponential growth of stereoscopic imaging in various applications, it has become very demanding to have a reliable quality assessment technique to measure the human perception of stereoscopic images. Quality assessment of stereoscopic visual content in the presence of artefacts caused by compression and transmission is a key component of end-to-end 3D media delivery systems. Despite a few recent attempts to develop stereoscopic image/video quality metrics, there is still a lack of a robust stereoscopic image quality metric. Towards addressing this issue, this paper proposes a full reference stereoscopic image quality metric, which mimics the human perception while viewing stereoscopic images. A signal processing model that is consistent with physiological literature is developed in the paper to simulate the behaviour of simple and complex cells of the primary visual cortex in the Human Visual System (HVS). The model is trained with two publicly available stereoscopic image databases to match the perceptual judgement of impaired stereoscopic images. The experimental results demonstrate a significant improvement in prediction performance as compared with several state-of-the-art stereoscopic image quality metrics.

This paper addresses the problem caused by motion vector coding dependencies on the error resilience performance of the emergent High Efficiency Video Coding (HEVC) standard. We propose a method based on the prediction dependency of motion vectors (MV) to select the most relevant ones for redundant coding with reduced overhead. The spatial dependencies are analysed in the encoder to prioritise the MVs that should be selected for redundancy, based on the number of subsequent dependent coding units. Then, a subset of prioritised MVs is transmitted as redundancy (referred to as side information in the paper), to reduce the use and propagation of mismatched MV predictions in case of transmission errors or data loss. The simulation results show that the proposed MV selection method can effectively identify the most relevant motion field, achieving improved error robustness with a reduced redundancy overhead. Exploiting only 30% of the generated MVs for redundancy, average quality gains of up to 1 dB are achieved compared to a uniform MV selection scheme, and up to 2 dB compared to the original HEVC standard with no redundant encoded information.

This paper proposes an immersive audio rendering scheme for networked 3D multimedia systems. The spatial audio rendering method based on wave field synthesis is particularly useful for applications where multiple listeners experience a true spatial soundscape while being free to move without losing spatial sound properties. The proposed approach can be considered as a general solution to the static listening restriction imposed by conventional methods, which rely on an accurate sound reproduction within a sweet spot only. The paper reports on the results of numerical analysis and experimental validation using various sound sources. It is demonstrated and confirmed that while covering the majority of the listening area, the developed approach can create a variety of virtual audio objects at target positions with very high accuracy. Subjective evaluation results show that an accurate spatial impression can be achieved with multiple simultaneous audible depth cues improving localization accuracy over single object rendering.

Streaming 3D multi-view video to multiple clients simultaneously remains a highly challenging problem due to the high-volume of data involved and the inherent limitations imposed by the delivery networks. Delivery of multimedia streams over Peer-to-Peer (P2P) networks has gained great interest due to its ability to maximise link utilisation, preventing the transport of multiple copies of the same packet for many users. On the other hand, the quality of experience can still be significantly degraded by dynamic variations caused by congestions, unless content-aware precautionary mechanisms and adaptation methods are deployed. In this paper, a novel, adaptive multi-view video streaming over a P2P system is introduced which addresses the next generation high resolution multi-view users' experiences with autostereoscopic displays. The solution comprises the extraction of low-overhead supplementary metadata at the media encoding server that is distributed through the network and used by clients performing network adaptation. In the proposed concept, pre-selected views are discarded at a times of network congestion and reconstructed with high quality using the metadata and the neighbouring views. The experimental results show that the robustness of P2P multi-view streaming using the proposed adaptation scheme is significantly increased under congestion.

Quality of Experience has recently become a paramount research topic in multimedia systems, especially in the emerging areas of high-definition and 3D video content. There has been a significant amount of research focusing on 3D content production, compression and delivery. However very little research has been dedicated to the emerging challenges in assessing user experience when interacting with the 3D video content. Interaction tasks such as pointing and selection are critical to the consumer's experience of the 3D video technology. This paper studies the impact of pointing modalities on the quality of interaction experience with stereoscopic 3D television. The conducted user study compares and evaluates three pointing modalities: standard mouse-based interaction, virtual laser pointer and hand gesture modality, using the ISO 9241-9 standard for multi-directional tapping task. The results suggest that the virtual laser pointer modality can provide better quality of interaction experience than other modalities in terms of user performance and user satisfaction.

This study presents a novel spatial audio coding (SAC) technique, called analysis by synthesis SAC (AbS‐SAC), with a capability of minimising signal distortion introduced during the encoding processes. The reverse one‐to‐two (R‐OTT), a module applied in the MPEG Surround to down‐mix two channels as a single channel, is first configured as a closed‐loop system. This closed‐loop module offers a capability to reduce the quantisation errors of the spatial parameters, leading to an improved quality of the synthesised audio signals. Moreover, a sub‐optimal AbS optimisation, based on the closed‐loop R‐OTT module, is proposed. This algorithm addresses a problem of practicality in implementing an optimal AbS optimisation while it is still capable of improving further the quality of the reconstructed audio signals. In terms of algorithm complexity, the proposed sub‐optimal algorithm provides scalability. The results of objective and subjective tests are presented. It is shown that significant improvement of the objective performance, when compared to the conventional open‐loop approach, is achieved. On the other hand, subjective test show that the proposed technique achieves higher subjective difference grade scores than the tested advanced audio coding multichannel.

Network Coding is a promising approach to increase network throughput and robustness to facilitate high volume traffic. Performing network coding in dynamic network structures requires transmitting coding coefficients for information sinks to decode network coded packets. Compared to the packet sizes used in practical networks, the size of coefficient vectors can be significant. This paper exploits the properties of small and medium sized networks and proposes a novel approach to minimise the coefficient vector size of network coded packets. Simulation results exhibit better compression of coefficient vectors over existing algorithms for small and medium sized networks.

The H.264/AVC coding standard has achieved significant coding gain by using flexible block sizes inter prediction. However, the complexity of inter-mode decision is extremely high. In this paper, a fast inter-mode decision algorithm is presented by exploiting residual homogeneity for both macroblock and sub-macroblock levels. In this algorithm, the residual homogeneity indicators calculated in three directions are used to identify the block's homogeneous characteristic. Normalized motion vector activity is also considered to improve the performance. The experimental results illustrate that the proposed algorithm achieves about 49% of computational complexity saving in terms of encoding time, while with negligible quality degradation, compared to H.264/AVC reference software JM.

The recent proliferation of stereoscopic three dimensional (3D) video technology has fostered a large body of research into 3D video capture, production, compression and delivery. However, little research has been dedicated to the design practices of stereoscopic 3D video interaction. Interaction tasks such as pointing and selection are critical to the consumer's experience of the 3D video technology. This paper presents investigation of pointing modalities in the context of stereoscopic 3D television (TV). Adopting the ISO 9241-9 standard for multi-directional tapping task, the conducted user study compares and evaluates three pointing modalities: standard mouse-based interaction, virtual laser pointer implemented using Wiimote, and hand movement modality using Kinect. The results suggest that the virtual laser pointer modality is more advantageous than other modalities in terms of user performance and user comfort. In addition, this paper discusses the impact of disparity levels on the pointing tasks.

The quality assessment of impaired stereoscopic video is a key element in designing and deploying advanced immersive media distribution platforms. A widely accepted quality metric to measure impairments of stereoscopic video is, however, still to be developed. As a step toward finding a solution to this problem, this paper proposes a full reference stereoscopic video quality metric to measure the perceptual quality of compressed stereoscopic video. A comprehensive set of subjective experiments is performed with 14 different stereoscopic video sequences, which are encoded using both the H.264 and high efficiency video coding compliant video codecs, to develop a subjective test results database of 116 test stimuli. The subjective results are analyzed using statistical techniques to uncover different patterns of subjective scoring for symmetrically and asymmetrically encoded stereoscopic video. The subjective result database is subsequently used for training and validating a simple but effective stereoscopic video quality metric considering heuristics of binocular vision. The proposed metric performs significantly better than state-of-the-art stereoscopic image and video quality metrics in predicting the subjective scores. The proposed metric and the subjective result database will be made publicly available, and it is expected that the proposed metric and the subjective assessments will have important uses in advanced 3D media delivery systems.

With the increasing popularity of online video streaming and broadcasting applications, video transmission has become the most bandwidth consumed application in the Internet. Unless resource utilisation is further optimised, existing networks would not be able to accommodate the forecasted demand for high volume video traffic. Random Linear Network Coding (RLNC) is a promising approach to increase the network throughput and robustness in order to facilitate high volume traffic. However, the performance gain obtained using RLNC decreases when the network packet loss rate increases. In response, this paper proposes to incorporate RLNC with a combined interleaving and a coding technique to improve the effectiveness of generating redundant packet combinations over classical RLNC to enhance the robustness against packet losses. Simulation results illustrate that the proposed method improves the probability of recovering lost data over the classical RLNC and the video transmission simulation results exhibit up to 5dB increase in PSNR and up to 14% increase in VQM utilising the same network resources at the same network conditions.

A scheme named index assignment-based channel coding (IACC) has been developed for resilience of speech/audio codecs against the bit errors commonly experienced in wireless channels. Although IACC is a type of joint source channel coding, it does not intervene with the source codec design. The proposed scheme takes into account source characteristics and adjusts the amount of coding according to the sensitivity of different values of the source parameters. It is shown that source characteristics play an important role in the performance of IACC. The performance of IACC has been evaluated by applying it to parameters generated by adaptive multi-rate wideband (AMR-WB+) audio codec. A method for perceptual training of IACC codes is also proposed. The results demonstrate that the performance of IACC and IACC concatenated with convolutional coding can be superior to that of conventional convolutional coding at high and moderate bit error rates, respectively.

Three different advanced adaptation techniques for improving the video perception of users are proposed in this paper. The proposed techniques exploit different adaptation decision-taking and adaptation approaches to adapt particular core parameters while considering diverse contextual information and constraints to achieve improved video perception of users. The first proposed technique employs a utility-based adaptation approach to perform adaptation operations on spatial resolution, frame rate, and quality scalability parameters according to the content-related contextual information (i.e., motion activity and structural feature) while fulfilling network bandwidth and terminal display size constraints. Using this technique, video contents can be adapted with the scalability parameters best fitting users' and contextual constraints' needs to achieve improved video perception. The second technique relies on prioritizing key frame, non-key frame, and temporal layer parameter-related network abstraction layer units to adapt video contents to satisfy network bandwidth constraint. The rate-distortion performances of adapted video contents can be improved by utilizing this technique in adaptation operations both in terms of bit rate of adapted video contents and video perception of users. The third technique is based on adapting the bit rate of 3-D video contents according to the changes in ambient illumination of the viewing environment. The adaptation results evaluated by either subjective or objective quality assessment techniques prove that all of the proposed techniques are efficient to improve the video perception of users.

Spatial hearing enables translation of an auditory scene into a perceived 3-D image by interpreting the acoustic cues related to the sounding objects, their locations, and the physical characteristics of the space. Spatial audio production requires multichannel audio signals in order to convey this information and increase the realism of a real or virtual environment for applications such as the home entertainment, virtual reality, and remote collaboration. As demand to spatial audio continues to expand, efficient coding of multichannel audio content becomes more and more important. This paper provides an overview of some well-known multichannel audio coding techniques and presents a new coding framework for improving the objective fidelity of the decoded signals. A closed-loop encoding system based on analysis-by-synthesis (AbS) principle applied on the MPEG surround (MPS) architecture is described. Comparison results are presented, which show that significant improvements can be achieved with a closed-loop system instead of the conventional open-loop system.

Three-dimensional (3-D) displays provide a more realistic experience of entertainment by providing its viewers an added sensation of depth by artificially exploiting light rays to stimulate certain depth cues in the human visual system, especially binocular stereopsis. Due to its close relationship with human visual perception, mass market deployment of 3-D displays will be significantly dependant upon addressing the related perceptual factors such as visual comfort. In order to address the perceptual factors, it is very important to understand how humans experience depth on 3-D displays and how sensitive they are for different depth cues. In this paper, the sensitivity of humans for different depth cues is analyzed as applicable to 3-D viewing on stereoscopic displays. Mathematical models are derived to explain the just noticeable difference in depth (JNDD) for three different depth cues, namely binocular disparity, retinal blur, and relative size. Extensive subjective assessments are performed on a stereoscopic display with passive polarized glasses and on an auto-stereoscopic display to validate the mathematical models for JNDD. It is expected that the proposed models will have important use cases in 3-D display designing as well as 3-D content production.

Delivery of 3D immersive entertainment to the home remains a highly challenging problem due to the large amount of data involved, and the need to support a wide variety of different displays. Support of such displays may require different numbers of views, delivered over time varying networks. This requires a delivery scheme featuring scalable compression to adapt to varying network conditions, and error resiliency to overcome disturbing losses in the 3D perception. Audio and video attention models can be used in designing an optimal content-aware compression and transmission scheme, by prioritizing the most visually important areas of the video. This paper gives an overview of a content-aware, scalable multi-view audiovisual entertainment delivery framework. Results are shown to evaluate the kinds of error robustness improvements that could be seen using such a system.

Interest in 3D video visualization systems is a ever growing field. Such areas include the provision of 3D content to users thus opening the exploration of 3D video communication and transmission. To address communication and transmission one must consider error resilience. Multiple Description Coding (MDC) can provide a robust video communication over wireless networks. However it can introduce high levels of redundancy. In this paper, we propose a scalable MDC architecture using motion vector (MV) encoding for 3D video. Experimental results show that the proposed algorithm can improve the frame quality by up to 2dB over a pixel based interpolation scheme with residual coding while significantly reducing the bit rate compared to a pixel and motion interpolation schemes.

The most distinguishing feature of 3D display systems, compared to their traditional 2D counterparts, is their ability to provide an additional perception of depth to its viewers. Thus, the mechanisms behind human depth perception play a significant role in 3D video systems. While there has been significant amounts of research carried out to understand human depth perception, in the areas of physiology and psychology, its applicability to 3D display systems is seldom spoken. Understanding the mechanisms of depth perception is of utmost importance to the development of 3D video technologies that are heavily based on exploitation of human perception. In this paper, it is explained with the aid of existing physiological and psychological models how humans perceive depth in 3D video displays. Based on these explanations a mathematical model is derived to explain the just noticeable difference in depth (JNDD) as perceived by a viewer, watching 3D video. The derived model is experimentally validated on an auto-stereoscopic display. This model is expected to be useful in both 3D content productions as well as in 3D content processing and compression.

Free viewpoint video enables the visualisation of a scene from arbitrary viewpoints and directions. However, this flexibility in video rendering provides a challenge in 3D media for achieving spatial synchronicity between the audio and video objects. When the viewpoint is changed, its effect on the perceived audio scene should be considered to avoid mismatches in the perceived positions of audiovisual objects. Spatial audio coding with such flexibility requires decomposing the sound scene into audio objects initially, and then synthesizing the new scene according to the geometric relations between the A/V capturing setup, selected viewpoint and the rendering system. This paper proposes a free viewpoint audio coding framework for 3D media systems utilising multiview cameras and a microphone array. A real-time source separation technique is used for object decomposition followed by spatial audio coding. Binaural, multichannel sound systems and wave field synthesis systems are addressed. Subjective test results shows that the method achieves spatial synchronicity for various viewpoints consistently, which is not possible by conventional recording techniques.

In this letter, a new bit rate adaptation decision taking technique is proposed based on the observations that ambient illumination context has an effect on the perceived video quality. Motion activity and structural complexity characteristics of a video content are utilized as the content-related contexts in the proposed technique. Experimental results demonstrate that a significant amount of bit rate can be saved while maintaining the perceptual quality same by adapting the video content according to different ambient illumination conditions using this technique. Subjective assessment results show that the proposed adaptation technique is capable of exploiting the changes in ambient illumination level for bit rate adaptation without sacrificing the perceived visual quality.

In this paper, a new morphing algorithm for transient sounds is introduced. Input sounds are first projected onto orthogonal bases from which intermediate or novel sounds can be generated. The proposed algorithm uses a shift invariant version of discrete wavelet transform and the singular value decomposition (SVD) to represent the input sound signals over a set of orthogonal bases. Interpolation is carried out between the weight vectors from the SVD to produce a new weight vector used for synthesising a new set of wavelet coefficients. The morphed sound is generated by taking the inverse discrete wavelet transform of the combined weighted bases. The proposed algorithm not only generates a range of new sounds, but also represents the input sounds in a more compact fashion.

This article presents a technique that uses the intensity vector direction exploitation (IVDE) method for exhaustive separation of convolutive mixtures. While only a four-element compact sensor array is used, multiple channels for all possible source directions are produced by exhaustive separation. Singular value decomposition (SVD) is then applied to determine the signal subspace and the directions of the local maxima of the signal energy. This information is then used to select the channels containing the individual sources. While the original IVDE method requires the prior knowledge of the directions of sources for separation, the present method eliminates this need and achieves fully blind separation. Performing SVD at a post-processing stage also improves the sound quality. The method has been tested for convolutive mixtures of up to four sources and typical separation performances are given.

In this paper we propose a novel video object edge adaptive upsampling scheme for application in video-plus-depth and Multi-View plus Depth (MVD) video coding chains with reduced resolution. Proposed scheme is for improving the rate-distortion performance of reduced-resolution depth map coders taking into account the rendering distortion induced in free-viewpoint videos. The inherent loss in fine details due to downsampling, particularly at video object boundaries causes significant visual artefacts in rendered free-viewpoint images. The proposed edge adaptive upsampling filter allows the conservation and better reconstruction of such critical object boundaries. Furthermore, the proposed scheme does not require the edge information to be communicated to the decoder, as the edge information used in the adaptive upsampling is derived from the reconstructed colour video. Test results show that as much as 1.2 dB gain in free-viewpoint video quality can be achieved with the utilization of the proposed method compared to the scheme that uses the linear MPEG re-sampling filter. The proposed approach is suitable for video-plus-depth as well as MVD applications, in which it is critical to satisfy bandwidth constraints while maintaining high free-viewpoint image quality.

This paper presents a content prioritization technique for perceptually enhanced communication of 3D video data over wireless local area network. The prioritisation is performed on the basis of a novel segmentation technique, which utilizes the depth information of 3D content for separating individual objects of the sequence. These objects are then encoded and transmitted separately under different protection levels depending on their perceived importance for perceptual visual quality. The algorithm performance is demonstrated for object based MPEG-4 video transmission over home network and is compared with a basic prioritisation scheme based on splitting a sequence into even and odd frames and encoding them separately with equivalent prioritization as in the proposed scheme. The results demonstrate significant improvement in subjective quality with the proposed scheme.

In this paper, an Unequal Error Protection (UEP) scheme for the transmission of 3-D (Three - Dimensional) video over WiMAX communication channel is proposed. The colour plus depth map stereoscopic video is coded with backward compatibility using a Scalable Video Coding (SVC) architecture, where users with conventional video decoders/receivers can receive the conventional 2-D (Three - Dimensional) video stream whereas users with SVC decoders/receivers and necessary 3-D video displays may render 3-D video. The proposed error protection scheme is based on the perceptual importance of the coded 3-D video components. The UEP method allocates more protection to the colour video packets than the depth map packets in order to receive good quality 2-D/3-D video. The protection levels are assigned through allocating differentiated transmission power for colour and depth map video packets during transmission. On the fly power allocation is based on the estimated distortion of the colour image sequence. The objective and perceptual quality evaluations show that the proposed UEP scheme improves the quality of 2-D video while achieving pleasing quality for 3-D viewers.

Distributed video coding (DVC) is an emerging video coding technology that utilises the distributed source coding principles to build very low cost video encoders, yet with remarkable error resilience. In the most common DVC framework, the reconstruction function plays a vital role that has a direct impact on the output video quality. In this study, a novel algorithm is proposed for the reconstruction function, particularly focusing on a unidirectional DVC architecture. The proposed technique exploits the variations of the bit error rate of the Wyner–Ziv decoded bit stream and the assumed noise model in the side information stream. The simulation results show that the proposed algorithm yields a significant improvement of the objective and subjective video quality at no additional bit rate cost.

In this letter, a new method is proposed for multiview depth map compression. It is intended to skip some parts of certain depth map viewpoints without encoding and to just predict those skipped parts by exploiting the multiview correspondences and some flags transmitted. It is targeted to save the bit rate allocated for depth map sequences to a great extent. Multiview correspondences are exploited for each skipped depth map frame by making use of the depth map frames belonging to neighboring views and captured at the same time instant. A prediction depth map frame is constructed block by block on a free viewpoint qualitywise selective basis from a couple of candidate predictors generated through the implicit and explicit usage of the 3-D scene geometry. Especially at lower bit rates, dropping higher temporal layers of certain depth map viewpoints and replacing them with corresponding predictors generated using the proposed multiview aided approach save a great amount of bit rate for those depth map viewpoints. At the same time, the perceived quality of the reconstructed stereoscopic videos is maintained, which is proved through a set of subjective tests.

This letter presents an improved decoding algorithm for distributed video coding (DVC) for enhanced performance over error prone wireless channels with multipath fading. The effects of the channel errors on both Wyner-Ziv and key frame bit streams are considered and novel noise models are proposed together with the associated modifications to the decoding algorithm. Simulations are performed using a W-CDMA wireless channel and the results are analyzed to determine the effect of each individual modification. State-of-the-art H.264/AVC is also considered under similar conditions for the comparison. The simulation results show that the proposed modifications provide a significant improvement in the DVC codec performance under unfavorable channel conditions.

The transmission of voice over IP networks is heavily affected by packet losses. An increasingly popular method to increase the error resilience of these systems is the use of multiple description coding (MDC). However, the MDC techniques commonly used tend to add a significant amount of redundancies, which are not always easy to use optimally. In this paper, we propose a simple vector quantisation scheme to maximise MDC performance, and study several factors affecting its performance under various error conditions. The results show that it is possible to obtain good performance under packet loss conditions, while using only limited amounts of redundancy.

The work presented in this paper introduces a method for efficient adaptability of video summaries to spatial requirements defined by display size, user’s needs and channel limitations. By utilising compressed domain features and an efficient contour evolution algorithm, a scale space of temporal video descriptors is generated, enabling dynamic video summarisation in real-time. The summary is laid out utilising an unsupervised robust spectral clustering technique and a fast discrete optimisation algorithm. Results show excellent scalability of the video summarisation interface and highly improved efficiency of summary generation.

This paper presents a novel channel optimisation scheme that enhances the quality of object based video, transmitted over a fixed bandwidth channel. The optimisation methodology is based on an accurate modelling of video packet distortion at the encoder. Video packets are ranked according to their expected distortion and are then mapped to one of a number of different priority radio bearers. In the proposed scheme the video compression technique uses motion compensated prediction and video frames are split into a number of video packets. The algorithm performance is demonstrated for object based MPEG-4 video transmission over a UMTS/FDD system. The results demonstrate that the performance gain achieved with the proposed scheme can reach 2 dB, compared with the equal error protection scheme for video transmission over a fixed bandwidth channel.

Source coding by exploiting the temporal and spatial correlations in an input video stream is well established in conventional video coding. However, when distributed video coding (DVC) is concerned, shifting the source coding from the encoder to the decoder made this a more complicated and possibly a sub optimal process. Exploiting the temporal correlations has so far been attempted utilizing the key frames, by motion compensation using the interpolated (or extrapolated) motion field. However, exploiting the spatial correlations within the DVC framework is still an unsolved problem. In this paper, we propose to exploit both spatial and temporal correlations to generate the side information at the decoder. The proposed architecture involves a bit plane level side information refinement mechanism extracting both temporal and spatial information available at the decoder. Simulation results show that, the proposed coding scheme can achieve about 40% bit rate saving over 3D refinement algorithm proposed in [1] which is utilized as the basis for implementing the proposed technique.

In this paper, an unequal power allocation (UPA) scheme for the transmission of SVC (scalable video coded) packets, over WiMAX communications channels is proposed. Transmission power is an important resource in wireless links, and its dynamic use enhances the successful transmission of video packets. Specifically, the proposed UPA scheme improves the decoded video quality by jointly optimizing the source rate allocated to encode the video and the power allocated to transmit the video packets. Base layer packets are allocated more power compared to enhancement layer packets, ensuring the efficient use of the available power budget. For fast, on the fly allocation of the power, lookup tables have been generated. The results show improvements in objective and subjective video quality.

This proposal presents a new error robust strategy for encoding redundant pictures for the H.264/AVC standard. The method is based on providing motion vectors as redundant data, i.e. providing extra protection to the motion information of the encoded stream. The proposed system is implemented based on the existing redundant coding algorithm of the scalable extension of H.264/AVC. The performance of the algorithm is evaluated using various objective quality measurements under both error free and error prone Internet protocol (IP) packet network environments. The proposed algorithm increases the bandwidth utilization with slight degradation in the primary picture quality for error free conditions, compared to the existing redundant coding method of JSVM (joint scalable video model). Furthermore, the simulation results under packet loss environments show that the proposed algorithm outperforms the existing redundant picture coding of JSVM.

In this paper, a new multi-view coding (MVC) scheme is proposed and evaluated. The scheme offers improved low-delay view random access capability and at the same time comparable compression performance with respect to the reference multi-view coding scheme currently used. The proposed scheme uses the concept of multiple-resolution view coding, exploiting the trade-off between quantization distortion and downsampling distortion at changing bit-rates, which in turn provides improved coding efficiency. Bi-predictive (B) coded views, used in the conventional MVC method, are replaced with predictive coded downscaled views, reducing the view dependency in a multi-view set and hence reducing the random view access delay, but preserving the compression performance at the same time. Results show that the proposed method reduces the view random access delay in a MVC system significantly, but has a similar objective and subjective performance with the conventional MVC method.

Stereoscopic video is one of the simplest forms of multi view video, which can be easily adapted for communication applications. Much current research is based on colour and depth map stereoscopic video, due to its reduced bandwidth requirements and backward compatibility. Existing immersive media research is more focused on application processing than aspects related to transfer of immersive content over communication channels. As video over packet networks is affected by missing frames, caused by packet loss, this paper proposes a frame concealment method for colour and depth map based stereoscopic video. The proposed method exploits the motion correlation of colour and depth map image sequences. The colour motion information is reused for prediction during depth map coding. The redundant motion information is then used to conceal transmission errors at the decoder. The experimental results show that the proposed frame concealment scheme performs better than applying error concealment for colour and depth map video separately in a range of packet error conditions.

New techniques for representing 3D video are often realised using monoscopic video and associated per-pixel depth information. While the compression of both monoscopic and depth video channels can be achieved using traditional video coding techniques, in this paper, specific properties of the depth channel are exploited to further compress the depth information. More precisely, enhanced compression of the depth channel is achieved by using a highly flexible motion model, binary partition tree (BPT) which enables adaptive partitioning of the depth frames leading to better rate-distortion optimisation for inter-frame prediction. Comparing to a conventional approach, limited variable size block (LVSB) which enables only limited frame partitioning options, the presented approach brings a gain of up to 70% in bitrate reduction for the depth information.

Due to a high demand for efficient video summarisation and video adaptation technologies, this paper focuses on utilisation of compressed domain feature extraction and hierarchical analysis of motion information in scalable video in order to generate intuitive visual summaries. By combining the analysis of inherently hierarchical motion activity measure and a fast geometrical curve simplification algorithm, a set of the most representative key-frames is generated in a very fast and robust manner. The experimental results show good subjective representation, while the method efficiency enables fast generation of summaries of large-scale video repositories.

Automatic video object segmentation based on spatial-temporal information has been a research topic for many years. Existing approaches can achieve good results in some cases, such as where there is a simple background. However, in the case of cluttered backgrounds or low quality video input, automatic video object segmentation is still a problem without a general solution. A novel approach is introduced in this work, to deal with this problem by using depth information in the algorithm. The proposed approach obtains the initial object masks based on depth map and on motion detection. The object boundaries are obtained by updating object masks using a simultaneous combination of multiple cues, including spatial location, intensity, and edge, within an active contour model. The experimental result shows that this method is effective and has good output, even with cluttered backgrounds. It is also robust when the quality of input depth and video is low.

Digital waveguide mesh (DWM) models are numerical solvers for the wave equation in -dimensions. They are used for obtaining the traveling-wave solution in practical acoustical modeling applications. Although unstructured meshes can be used with DWMs, regular mesh topologies are traditionally used due to their implementation simplicity. This letter discusses the accuracy of first-order approximations to numerical derivatives on more general unstructured mesh topologies. The results are applied to structured, regular mesh topologies as used in DWM modeling. A comparison of 2-D and 3-D DWM topologies with respect to the accuracy of first-order approximations to numerical derivatives is presented.

Various techniques have previously been proposed for the separation of convolutive mixtures. These techniques can be classified as stochastic, adaptive, and deterministic. Stochastic methods are computationally expensive since they require an iterative process for the calculation of the demixing filters based on a separation criterion that usually assumes that the source signals are statistically independent. Adaptive methods, such as the adaptive beamformers, also exploit signal properties in order to optimize a multichannel filter structure. However, these algorithms need initialization and time to converge. Deterministic methods, on the other hand, provide a closed-form solution based on the deterministic aspects of the problem, such as the channel characteristics and the source directions. This paper presents a technique that exploits the intensity vector statistics to achieve a nearly closed-form solution for the separation of the convolutive mixtures as recorded with a coincident microphone array. No assumptions are made on the signals, but it is assumed that the source directions are known a priori. Directivity functions based on von Mises functions are designed for beamforming depending on the circular statistics of the calculated intensity vectors. Numerical evaluation results were presented for various speech and instrument sounds and source positions in two reverberant rooms.

Digital waveguide mesh (DWM) models offer a simple, accurate, time-domain, numerical solution of the wave equation. A specific case where such accurate and computationally simple solutions are needed is the acoustical modeling of open or closed volumes. It is possible to model 3-D propagation of waves in enclosures such as rooms using DWM models. Generally, idealized omnidirectional sources are used for obtaining the impulse response in the DWM. However, real-life sound sources are never completely isotropic, causing wavefronts with directional properties. This paper presents two methods to simulate analytical and empirical directivities in 3-D DWM models in the far-field. The first method is based on the direct excitation of the mesh with the velocity component of the directional source and is used to simulate analytical sources. The second method is based on the weighting of velocity components generated by an omnidirectional source at different octave-bands and is used to simulate sources with frequency-dependent empirical directivity functions. A simple interpolation method for obtaining a closed-form description of the directivity function from incomplete directivity data is also proposed. Simulation results are presented for two sources in an acoustical model of a rectangular room.

Various 3-D face synthesis techniques have been proposed and extensively used in many applications. Compared with others, single view-based face synthesis technology allows unsupervised 3-D face reconstruction without any offline operations. Although many algorithms have been published, automatic and robust single view-based 3-D face synthesis still remains unsolved. In contrast to other methods, the single view-based 3-D face synthesis algorithm conducted in this paper enables automated 3-D face synthesis from an arbitrary head-and-shoulder image with the complex background. The developed system first detects the face using Bayesian skin-tone classification based on only the chrominance component, Cr. Based on the detected face, a few salient facial features, such as the corners of the eyebrows and contours of the eyes, mouth, and chin are in turn extracted using variant algorithms, including a dynamic chin extraction mechanism that will be detailed in this paper. Then, face model adaptation consisting of both global and local adaptations is imposed, according to geometric information provided by the extracted facial features. Finally, the 3-D specific face is synthesized using the adapted 3-D face model with a texture map directly derived from the input face image, followed by the implementation of facial animation using this synthesized face.

In this paper we discuss the ease with which email can be used to breach confidence by the propagation of corporate secrets and intelligence, and propose an intelligent filtering system for outgoing emails aimed at preventing disclosures. We report on a number of experiments undertaken with a corpus of over half a million Enron emails and the use of a variety of techniques from the field of Corpus Linguistics for reducing the number of false alarms produced by naive keyword filtering systems, and discuss the results in detail. We also give due consideration to the danger of missing messages that should have been prevented from propagation.

This paper proposes a framework for a virtual classroom application based on a Virtual Collaboration System (VCS), which is being developed under the VISNET II Network of Excellence (NoE)1, and discusses adaptation technologies that enable seamless access to classroom sessions while intellectual property and digital rights are managed. The proposed virtual classroom framework enables academic institutions to conduct their collaborative lecture series, to which registered students will be able to attend remotely and interactively over the Internet. Furthermore, the general public may also follow the classroom sessions under certain restrictions imposed by the participating institutions. In order to facilitate seamless access to a heterogeneous audience that is composed of users with various preferences and privileges accessing the classroom sessions over different network infrastructures and using terminal devices with diverse capabilities, context-aware content adaptation is required to meet constraints imposed by the usage context and enhance the quality of the user experience. Thus, this paper describes the concepts and functionalities of a context-aware content adaptation platform that suits the requirements of such multimedia application scenarios. This platform is able to consume low-level contextual information to infer higher-level contexts, and thus decide the need for and type of adaptation operations to be performed upon the content. In this way, it is aimed to meet usage constraints while also satisfying restrictions imposed by the Digital Rights Management (DRM) to govern the use of protected content.

Fractional-delay filter is the general name given to filters modelling non-integer delays. Such filters have a flat phase delay for a wide frequency band, with the value of the phase delay approximating the fractional delay. A maximally-flat delay IIR fractional-delay filter can be obtained by the Thiran approximation. A simple and efficient method for obtaining filters modelling intermediate fractional delays from two Thiran fractional-delay filters is proposed. The proposed method allows continuously modifying the fractional delay. Computational complexity of the proposed method is discussed. A practical application of the method in model-based sound synthesis is given as an example

CELP coders, such as G.729, are often used in VoIP systems as they offer good speech quality in the absence of packet losses. However, their reliance on long-term prediction causes propagation of errors across speech frames, and therefore makes CELP coders more sensitive to packet losses. Sinusoidal coders on the other hand do not rely on long-term prediction, and may be a good alternative for VoIP due to their higher resilience to packet losses. In this paper a comparison is made between CELP and sinusoidal coders in a VoIP application. A packetisation scheme based on multiple description coding (MDC) applied to the sinusoidal coder is presented. The results show that under typical VoIP operating conditions, the sinusoidal coder based systems can outperform CELP based systems at equal bit rate, especially for high packet loss rates

Applications such as transaction authentication may require speaker recognition systems to operate on compressed speech transmitted over mobile phone networks. However, speech compression degrades speech quality, and hence causes a reduction in recognition performance. It has been shown that the classic technique for extraction of line spectral frequency (LSF) parameters in speech coders is prone to aliasing distortion. The use of a low-pass filtering on upsampled LSF vectors has been shown to alleviate this problem, therefore improving speech quality. In this paper, the effect of this non-aliased LSF (NA-LSF) extraction method on speaker recognition performance is observed using GSM-EFR coded speech. When the NA-LSFs are used in the coder the verification performance loss introduced by the GSM-EFR is reduced, producing similar performance to uncoded speech

Scalable shape encoding is an important requirement of highly scalable object-based video coding. In this paper, a new scalable vertex-based shape coding scheme is proposed that uses temporal prediction. During shape coding of a video object, the object shape in the first frame is scalable intra-coded using the method described in M. Hu et al., (2004). For scalable shape coding of subsequent frames, temporal prediction is conducted during the coding of coarser layers. Contour matching in the curvature scale space domain is conducted in order to get higher matching accuracy. Experimental results show that the proposed scalable shape coding scheme can achieve better R-D performance than existing predictive shape coding methods and CAE in MPEG-4.

Scalable shape encoding is one of the important steps to achieving highly scalable object-based video coding. In this paper, a new scalable vertex-based shape intra-coding scheme has been described. To improve the encoding performance, we propose a new vertex selection scheme, which can reduce the number of approximation vertices. We also propose a new vertex encoding method, in which the information on the coarser layers and statistical entropy coding are exploited for high encoding efficiency. Experimental results show that the proposed scheme can provide 25-60% gain over the scalable encoding method in Buhan Jordan et al. (1998). For some sequences, it can achieve 5-10% gain over the conventional non-scalable vertex-based coding method (O'Connell (1997)) in bit rate, at the price of additional complexity.

We present an efficient scalable object contours tracking algorithm and its application for video segmentation. It can track the contour of both rigid and nonrigid objects, even with partial occlusion. For each object, motion information is first used to predict the object contour. Next, texture information along the object contour is used to refine the predicted contour. Finally, active snake is used to snap the detected contour to the real object with high accuracy. The proposed energy terms in active snake model are normalized colour gradient and foreground/ background region properties along the contour. Experimental results illustrate the performance of the proposed tracking algorithm, which is able to track the object boundaries with large motion and partial occlusion.

The quality of low bit-rate speech coders is reduced at transitions where speech spectral characteristics vary significantly, as usual speech parameter interpolation assumptions fail to correctly model such variations. This paper presents a joint quantisation-interpolation algorithm for coding of LPC parameters in pitch synchronous speech coders to model the rapidly evolving parameters. In this technique a number of sets of pitch synchronous LPC parameters, corresponding to a frame of speech, are jointly coded by coding two reference sets of LSF's and an interpolation trajectory. Coding an interpolation function allows the parameters to vary within the set. The proposed joint quantisation-interpolation coding of the pitch synchronous LSF is evaluated by comparison with time synchronous extraction and linear interpolation. It is also compared with linear interpolation between sets of pitch synchronous LSF's. Comparison results show that the joint quantisation-interpolation method reduces the average spectral distortion when compared to fixed interpolation. The proposed quantiser was included in to the PS-SBLPC coder and informal listening tests carried out. The synthesised speech was found to be of better quality when joint quantisation-interpolation is used.

A novel fully comprehensive mobile video communications system is proposed. The system exploits the useful rate management features of video transcoders and combines them with error resilience for the transmission of coded video streams over general packet radio service (GPRS) mobile-access networks. The error-resilient video transcoding operation takes place at a centralized point, referred to as a video proxy, which provides the necessary output transmission rates with the required amount of robustness. With the use of this proposed algorithm, error resilience can be added to an already compressed video stream at an intermediate stage at the edge of two or more different networks through two resilience schemes, namely the adaptive intra refresh (AIR) and feedback control signaling (FCS) methods. Both resilience tools impose an output rate increase which can also be prevented with the proposed novel technique. Thus, the presented scheme gives robust video outputs at near target transmission rates that only require the same number of GPRS timeslots as non-resilient schemes. Moreover, an ultimate robustness is also accomplished with the combination of the two resilience algorithms at the video proxy. Extensive computer simulations demonstrate the effectiveness of the proposed system.

This paper presents the behavioural mechanism of a statistical model-based voice activity detector (VAD), featuring a likelihood ratio test for the activity decision. From investigation of the VAD, it is found that detection errors could occur frequently at speech offset regions because of the delay term in the decision-directed parameter estimator, employed for the estimation of an unknown parameter of the likelihood ratio. Hence, this paper proposes a smoothed likelihood ratio so as to alleviate the detection errors at the offset region. Objective test results show that the proposed scheme is useful for achieving a considerable performance improvement for the VAD. Additionally, the proposed VAD gives detection performances superior to G.729B VAD and comparable with the AMR VAD option 2.

This paper presents a hybrid coder with a new phase model to synchronize harmonic and waveform coded segments, with a target bit rate of 4 kbps. The coder also employs a new technique based on analysis by synthesis to distinguish between stationary and transitional segments. Harmonic excitation is synchronized with the LPC residual by transmitting the location of the pitch pulse closest to the frame boundary and a phase value that represents the shape of the corresponding pitch pulse. The performance of this phase model and the classification technique is evaluated using a hybrid coder. The coder has three modes: scaled white noise excitation colored by LPC for unvoiced, ACELP for transitions, and harmonic excitation for stationary segments. Subjective listening tests show that the coder produces good quality speech and the switching between the modes is transparent.

Optimising delivery of video codecs such as MPEG-4 is vital to ensure that acceptable quality can be offered to 3G network customers. The error prone nature of mobile channels means that the video codec that is employed must be fairly robust. In the past, such research has required alteration to existing standards. Given the imminent implementation of 3G technologies, changing the standards is not a convenient option. This paper presents a technique for altering MPEG-4 encoding parameters to increase the error robustness of the output bitstream. It exploits the fact that frames with a high degree motion are often more sensitive to error than those with low amounts of motion. Using a simple model, the video packet length and number of adaptive intra refresh (AIR) blocks are varied according to the amount of motion in a frame. Simulations using a GPRS channel are presented to confirm the benefits of the proposed scheme.

The integration of fixed and wireless networks have expanded the range in which speech and audio coders were designed to operate. As a result, current research is focusing on a number of developments including algorithms which are able to adapt to different transmission environments and to operate under multiple constraints of bit rate, complexity, delay, robustness to bit errors and diversity of input signals. In this paper, we propose a single coding algorithm for compressing wideband speech and audio signals (0-8kHz) operating with scalable bit rates and with low delay. The algorithm is based on the backward-adaptive linear predictive coding (BA LPC) technique in conjunction with an efficient closedloop optimised excitation structure consisting of sparse pulses of ternary values. The output bit rates range from 17 to 68kb/s. The scalability feature is achieved by means of discrete quantisation layers representing various levels of enhancements of the base-line coder and also flexibility in terms of complexity and bit allocation requirements depending on the particular application and on the network resources. An evaluation of the performance of the coder operating at 17kb/s is carried out using the G.722 standard as a reference.

The past two decades have witnessed a rapid expansion within the telecommunications industry. This growth has been primarily motivated by the proliferation of digital communication systems and services which have become easily available through wired and wireless systems. Current research trends involve the integration of speech, audio, video and data channels into true multimedia communications over fixed and mobile networks. However, while the available bandwidth in wired terrestrial networks is relatively cheap and expandable, it becomes a limited resource in satellite and cellular-radio systems. In order to accommodate an ever growing number of users while maintaining high quality and low operational costs, it is necessary to maximise spectral efficiency. This has given rise to the development of high rate compression techniques with the ability to adapt to a broad class of input signals and to varying network resources. The research carried out in this thesis has mainly focused on the design of a single algorithm for compressing speech and audio signals sampled at different rates. The algorithms are based on the analysis-by-synthesis linear prediction coding (AbS-LPC) scheme, which has been widely employed in various speech coding standards. However, this bit rate reduction technique is based on the speech production mechanism and as such provides a rigid structure which presents a major limitation for audio coding. In order to improve the audio quality at low rates and to compensate for the errors incurred by the linear prediction during segments of high transitions, the algorithms employ an efficient pulse excitation structure which represents the short innovation sequences with sparse unit magnitude pulses. The scheme proposed for the compression of telephone bandwidth speech and audio signals at 12kb/s achieves similar quality to the G.728 coder at 16kb/s and higher audio quality than the GSM-EFR standard at 12.2kb/s. Wideband speech and audio coding schemes have been designed using both the fullband approach at bit rates of 17 and 19kb/s and also the split band technique at a bit rate of 20kb/s. The perceptual quality is comparable to the G.722 coder operating at 48kb/s. The subband decomposition technique is also adapted to code speech and audio signals sampled at 32kHz. The quality of the coder at 28kb/s is similar to the quality achieved by the MP3 coder at 32kb/s. The algorithm also provides bandwidth and bit rate scalability ranging from 12 to 64kb/s, making it ideal for deployment in rate-adaptive communication systems.

A novel synchronization technique is proposed for hybrid coders employing harmonic and waveform coding. A new classification technique based on analysis by synthesis to distinguish between stationary and transitional segments is also proposed. Harmonic excitation is synchronized with the LPC residual by transmitting the location of the pitch pulse closest to the frame boundary and a phase value that represents the shape of the corresponding pitch pulse. A hybrid coder is designed to demonstrate the new techniques, which has three modes: scaled white noise excitation colored by LPC for unvoiced, ACELP for transitions, and harmonic excitation for stationary segments. Subjective listening tests show that the speech quality of the variable rate hybrid coder outperforms the quality of ITU G.723.1 coders, at maximum bit rates lower than those of G.723.1 coders.

From an investigation of a statistical model-based voice activity detector (VAD), it is found that the likelihood ratio defined in the VAD has a fundamental problem at the offset regions of the speech signals. Thus, we analyze the behavioural mechanism of the likelihood ratio, identify the reason for the unwanted phenomenon, and propose a solution based on a smoothed likelihood ratio. Objective test results show that the proposed method gives a significant improvement to the original VAD. Additionally, the improved VAD results in a performance that is even superior to G.729B and comparable to AMR VAD option 2.

In sinusoidal speech coding, the LP-spectral envelope is limited in its spectral accuracy if the order of the LP-model is not high enough. Thus the quantization of the residual spectrum of the low order LP-model may be desirable for good quality speech reconstruction. From the investigation of the magnitude of the LP-residual spectrum, it is found that the predictive coding scheme is useful for removing coding redundancy considerably. The problem of having a variable number of harmonics due to pitch changes can be alleviated by a length warping technique. Subsequently, the residual spectrum of the predictive coding is represented by mel-scale binary vector quantizer (MBVQ), which quantizes the residual spectrum by splitting harmonic bands of variable dimension into a fixed dimension, based on mel scale, and representing each element of the code vector as a binary value. The optimal code vector for the MBVQ can be derived by minimizing an error measure, defined as the weighted square-sum of the difference between original and synthesized spectral envelopes. From the performance evaluation, it is shown that the predictive-coded MBVQ with low order LP can obtain the effect of considerably high order LP-model. Additionally, the proposed method can be implemented with very low computational complexity in time and space.

In this paper we propose a multiple frequency harmonics model for analysis and synthesis of audio signals. The novelty of this model is that a composite sound can be represented by a few number (two or three) of frequency harmonics, each with time-varying fundamental frequency. We present the model and some key issues in tuning it. Results demonstrate that the model is valid and promising. It is convenient for time- and frequency-scale modifications and is of interest for low bit rate audio coding.

This paper presents a scalable audio format, called "multi-layer scalable LPC audio format", that addresses similar functionalities of MPEG-4. The format offers different levels of data rate and audio quality, and answers to the most important requirements of transmission and storage purposes, such as channel error robustness, cell loss robustness, low delay, and playback control. It operates in four modes. The first mode is based on a modified version of the LD-CELP algorithm, in which each 6 samples are represented by one single byte. In order to improve the signal-to-noise ratio (SNR), additional enhancement layers are embedded in the bit stream to allow higher quality at higher bit rates. The resultant bit rates are integer-multiple of 10.67 kbps. The other three modes use QMF splitting to two, four and eight subbands. These modes allow efficient representation of wideband audio and speech signals, and offer extension layers of 5.33 and 2.66 kbps. A simple and efficient header structure is embedded in the bitstream to allow the decoding process even in channel error conditions and even when the bitstream has been down-scaled somewhere during the transmission but has not been acknowledged to the decoder. Comparison results are conducted with respect to MPEG and ITU standards.

The European Telecommunications Standards Institute (ETSI) has launched a competition for a new mobile communications standard designed to provide better performance than the current GSM standard. This standard is to be called AMR for adaptive multi-rate: the source and channel coding rates can be adapted depending on the state of the channel, thus providing optimal balance between them at any time. The University of Surrey has submitted a candidate for this competition through the Mobile VCE. This candidate was the only one amongst eleven to use a vocoder in the half-rate GSM channel instead of a CELP based coder. The testing which took place as part of the first stage of the competition has shown that this candidate was among the best. This paper presents the system submitted for the half-rate channel as well as the results of the testing.

In previous studies of formant frequency discrimination, variation in the stimuli has mainly concerned the formant frequencies while other factors which may affect formant frequency discrimination have largely been ignored. In most studies, fundamental frequencies typical of adult male speakers have been used. In the study presented here, formant frequency discrimination as a function of fundamental frequency level was studied. Contrary to expectations, fundamental frequency level did not seem to affect formant frequency discrimination. This was true even when the fundamental frequency was higher than the tested formant frequency. The implication of these findings is that formant frequency discrimination must, at least partly, rely on some other mechanism than reconstruction of the vocal tract transfer function. It is suggested that listeners use differences in the amplitude of partials to discriminate between complex vowel like sounds. Asymmetries in the distribution of these amplitudes may also explain the corresponding asymmetries found in perception.

We describe a method for automatically learning a parser from labeled, bracketed corpora that results in a fast, robust, lightweight parser that is suitable for real-time dialog systems and similar applications.Unlike ordinary parsers, all grammatical knowledge is captured in the learned decision trees, so no explicit phrase-structure grammar is needed.Another characteristic of the architecture is robustness, since the input need not t pre-speci ed productions.Even without using speci c lexical features, we have achieved respectable labeled bracket accuracies of about 81 precision and 82 recall.Processing speed is more than 500 words per CPU second.We k eep the parameter space small in comparison to other statistically learned parsers by using only part-of-speech tags and constituent labels as features.Without any optimization, the decision trees consume only 6M of memory, making it possible to run on platforms with limited memory.The learning method is readily applicable to other languages.Preliminary experiments on a Chinese corpus which contains about 3000 sentences from Chinese primary school text have yielded results comparable to that for English.

LPC based speech coders operating at bit rates below 3.0 kbits/sec are usually associated with buzzy or metallic artefacts in the synthetic speech. These are mainly attributable to the simplifying assumptions made about the excitation source, which are usually required to maintain such low bit rates. A new LPC vocoder is presented which splits the LPC excitation into two frequency bands using a variable cut-off frequency. The lower band is responsible for representing the voiced parts of speech, whilst the upper band represents unvoiced speech. In doing so the coder's performance during both mixed voicing speech and speech containing acoustic noise is greatly improved, producing soft natural sounding speech. The paper also describes new parameter determination and quantisation techniques vital to the operation of this coder at such low bit rates.

This paper investigates the benefits of link adaptation according to the propagation conditions using multi-rate vocoders and channel coders. A multi-rate source and channel coding (MRC) system trades-off the source codec bit rate against channel codec bit rate according to the prevailing channel conditions. MRC systems continuously reconfigure transmission resources to optimally combat channel degradations and hence, maintain a reasonable quality of service even in marginal channel conditions. The aim is to maintain a satisfactory level of speech quality over a wide range of channel conditions with minimum resources. This paper shows the benefit of MRC system particularly in crowded cells of high channel interference.

Prototype waveform interpolation is one of the most efficient compression techniques for coding the speech signal at bit rates below 4 kb/s. Most of the PWI coders employ prototype waveforms of the linear predictive residual signal for coding purpose. In the latest PWI systems, decomposition methods are used to separate the voiced and unvoiced components of the prototype waveforms prior to coding. This has resulted in high quality speech at very low bit rates. This paper presents a novel combination of the multiband voicing analysis and PWI coding system in which the multiband analysis is exploited to identify the voiced and unvoiced spectral components of the prototype waveforms of the original speech signal. To produce a high quality synthetic speech, the energy variation of the original signal is recovered by transmitting its energy envelope. This method resulted in a high quality and low complexity coder operating at 2.55 kb/s.

This paper presents a novel video coding algorithm which exploits past frame statistics for entropy coding of the gain-shape VQ parameters. A subband transform has been used to decorrelate the motion compensation residual in order to overcome the block visibility problem. The vectors with samples from different subbands are quantised using a gain-shape VQ technique which allows effective bit rate control. When compared to the H.261 standard, the proposed algorithm has given higher PSNR values at lower bit rates.

This paper presents a linear prediction (LP) based vocoder employing a novel technique which ensures smooth evolution of the synthetic speech waveform. In this coder, speech waveforms are considered as having a 'time envelope', the shape of which contains important perceptual information. By ensuring that the time envelope of the synthetic speech closely matches that of the original, natural sounding synthetic speech can be produced. Envelope matching may be achieved using a new, low complexity analysis by synthesis loop at the decoder which determines the synthetic excitation energy. The advantage over more traditional linear prediction vocoders is that the amplitude time envelope is preserved in addition to the spectral envelope, allowing the rapid amplitude transitions associated with onsets to be retained in the synthetic speech, resulting in a more intelligible output. Simply controlling the overall energy of the synthetic excitation is not sufficient to accurately control the synthetic speech energy. Small changes in linear prediction or pitch parameters due to quantization, for example, can cause variations in the synthetic speech amplitude, especially from one pitch cycle to the next resulting in noisy synthetic speech. The inclusion of an analysis by synthesis loop at the decoder ensures that the synthetic speech amplitude is independent of variations in the pitch period and LP parameters. This paper presents a complete vocoder scheme using time envelope matching, including details of techniques such as parameter interpolation, excitation pulse shaping and pitch tracking which have proven necessary to produce natural sounding synthetic speech at 2.4 kb/s and below.>

The authors present an 8 kb/s low delay speech coder which is based on the code excited linear prediction (CELP) algorithm. They concentrate on analyzing the problem areas in backward predictive systems and solutions to alleviate these problems. The authors report on a forward linear predictive coding adaptive scheme to eliminate such distortions. Another approach is to use a windowing technique in the frequency domain on the error signal in analysis-by-synthesis procedures. The adaptation of the long term predictor (LTP) coefficients is also considered in a backward and forward mode for differential pitch coding of the LTP lags. The role of the excitation gain update rate is also examined in providing higher quality and better adaptation of the gains for an efficient vector quantization procedure. An 8-kb/s coder with a buffering delay of 4 ms has been informally subjectively tested, and the latest test results indicate a performance equal to that of the GSM at worst case.>

Optimum quantization and code assignment schemes are described which minimize the subjective quality degradations introduced into the output speech of code excited linear predictive (CELP) coders by channel degradations. The background and basis for use of minimum redundancy for error control are examined. Greater emphasis is placed on adjustment of corrupted parameters to minimize subjective degradation rather than outright bit-by-bit error correction. These schemes are mostly tested on the CELP baseband coder, but they should be applicable to any linear predictive coder. They raise the bit rate of a 4.8-kb/s coder by about 12.5% and its MOS at 2*10/sup -2/ bit error rate by about 21.1% (scale 1-5).>

An improved CELP (code-excited linear prediction) baseband (BB) coder was tested using informal subjective listening tests. The overall bit rate of the coder was 4.7 kb/s. The subjective quality of speech scored just over three on a five-point mean opinion scale. Improvements are described that increase the score by about 0.35 over its original form. This means that in terms of quality the CELP BB coder at around 4.8 kb/s can replace the analog ACSSB operating even at 51 dB/Hz because in its original form it had subjective quality equivalent to ACSSB. To further investigate the performance of the coder the authors introduced random channel errors into the bit stream between the encoder and the decoder. At bit error rates of 10/sup -3/ and less, the coder did not need any forward error protection. When the bit error rate was around 10/sup -2/ an unexpected distortion occurred; the output energy level of the coder was very low whenever there were errors in the pitch filter or the optimum vector gain parameters. This was due to the new adaptation scheme used for the optimum vector gain. The authors solved this problem by adapting the optimum gain with the maximum block energy of the pitch filter memory or by simply using the energy of the whole of the pitch filter memory.>

A baseband coder is discussed and its subjective quality compared with code-excited linear prediction (CELP) and vector-quantized transform coding at various bit rates from 9.6 to 4.8 kb/s. It is also with that CELP-BB coding is capable of producing natural, intelligible and smooth speech at 2.4 kb/s. The code-book search in CELP-BB requires no convolution. The pitch filter memory is subtracted from the original baseband signal to form the reference signal, which is directly matched by the code-book sequences. Because most of the computation is performed on the baseband, which is decimated, and during code-book search only the pitch filter response is considered, the complexity of CELP-BB is very much less than that of CELP and its speech quality is not affected by LPC quantization error to such an extent as in CELP.>