Abstract In this work, MineStudio is introduced, a novel AI filmmaking framework designed to facilitate future creative collaborative networks. MineStudio uses a hybrid digitization approach that involves reconstructing 3D digital environments, capturing 2D live‐action performances, employing AI tools to generate synthetic images and videos, and compositing with AI assistance. This method effectively addresses the main challenges in the current AI video generation, including consistency, directability, and issues with human actions and interactions. MineStudio has been utilized to create pioneering AI films, such as the love story “Next Stop Paris” and the sci‐fi short film “Message in a Bot”, and has been recognized as a trailblazer in the AI filmmaking industry.

User-generated cinematic creations are gaining popularity as our daily entertainment, yet it is a challenge to master cinematography for producing immersive contents. Many existing automatic methods focus on roughly controlling predefined shot types or movement patterns, which struggle to engage viewers with the actor's circumstances. Real-world cinematographic rules show that directors can create immersion by comprehensively synchronizing the camera with the actor. Inspired by this strategy, we propose a deep camera control framework that enables actor-camera synchronization in three aspects, considering frame aesthetics, spatial action, and emotional status in the 3D virtual stage. Following rule-of-thirds, our framework first modifies the initial camera placement to position the actor aesthetically. This adjustment is facilitated by a weakly-supervised adjustor that analyzes frame composition via camera projection. We then design a GAN model that can adversarially synthesize fine-grained camera movement based on the actor's action and psychological state, using an encoder-decoder generator to map kinematics and emotional variables into camera trajectories. Moreover, we incorporate a regularizer to align the generated stylistic variances with specific emotional categories and intensities. The experimental results show that our proposed method yields immersive cinematic videos of high quality, both quantitatively and qualitatively. Live examples can be found in the supplementary video.

Advancements in wearable technology and their capacity to interpret user movements, transforming them into interactive actions in virtual environments, have sparked an increased demand for user flexibility within these spaces. A direct outcome of this growing trend is the imperative need for automated cinematography in expansive, open-world scenarios. Nevertheless, the task of interpreting these interactive sequences through automated cinematography in unconstrained environments involves significant computational challenges. In response to this, we introduce the Automated Adaptive Cinematography for Open-world Generative Adversarial Network (AACOGAN) -an innovative solution that addresses these issues. Contrary to traditional models, which require comprehensive prior knowledge about scenes, characters, and objects, AACOGAN identifies and models the relationships among user interactions, object positions, and camera movements during the process of user engagement. This novel approach allows the model to function effectively even in open-world scenarios riddled with numerous uncertain factors. In the experimental phase, we developed and employed theMineStory Dataset, designed specifically for automatic cinematography in open-world scenarios. We devised and implemented novel metrics that are more congruent with the distinctive features of open-world scenarios. These innovative metrics provide a more nuanced understanding of the performance and effectiveness of our proposed method. Experimental findings substantiate that AACOGAN significantly enhances automatic cinematography performance within open-world contexts, including an average augmentation of 73% in the correlation between user interactions and camera trajectories, and an increase of up to 32.9% in the quality of multi-focus scenes. Therefore, AACOGAN emerges as an efficient, and innovative solution for creating appropriate camera shots in a myriad of interactive motions in open-world scenarios. An exemplary video footage can be found athttps://youtu.be/pbSHF-uxomw.

With the rapid growth of IoT multimedia devices, more and more content is being delivered in multimedia forms which generally requires more effort and resources from users to create and could be challenging to streamline. In this article, we propose Text2Animation (T2A), a framework that helps to generate complex multimedia content, and animation, from the simple textual script input. By leveraging recent advances in computational cinematography and video understanding, the purposed workflow further reduces associated knowledge requirements dramatically for cinematography. By jointly incorporating the fidelity and aesthetic models, T2A jointly considers the comprehensiveness of the visual presentation of the input script and the compliance of generated video with given cinematography specifications. The virtual camera placement in a 3-D environment is mapped into an optimization problem that can be resolved by using dynamic programming to achieve the lowest computational complexity. Experimental results show that T2A can reduce the manual animation production process by around 74%, and the new optimization framework can improve the perceptual quality of the output video by up to 35%. More video footage can be found athttps://www.youtube.com/watch?v=MMTJbmWL3gs.

With the technology development, more and more Internet of Things (IoT) devices with displays are making “face-to-face” interaction through visualization a reality. To protect the privacy of users, communications can be represented through avatars and use audio-driven real-time speech animation. However, if audio is the only available input, the quality of the outcome relies heavily on real-time phoneme recognition, such as recognition accuracy and latency. This article introduces a novel deep-learning-based real-time phoneme recognition network (RealPRNet) scheme to leverage spatial and temporal patterns in the input audio data. With featured long short-term memory stack block and long short-term features, RealPRNet can achieve super performance in phoneme recognition. Our comprehensive empirical results show that compared to the state-of-the-art algorithms, RealPRNet can achieve 20% phoneme error rate (PER) improvement and 4% block error distance (BDE) improvement in the best case.

Media content has become a primary source of information, entertainment, and even education. The ability to provide video content querying as well as interactive experiences is a new challenge. To this end, question answering (QA) systems such as Alexa and Google Assistant have become quite established in consumer markets but are limited to general information and lack context awareness. In this paper, we propose Context-QA, a light-weight context-aware QA framework, to provide QA experiences on multimedia content. The context awareness is achieved through our innovative Staged QA Controller algorithm that keeps the search for answers in the context most relevant to the question. Our evaluation results show that Context-QA improves the quality of the answers by up to 49% and uses up to 56% less time compared to the conventional QA model. Subjective tests show Context-QA improved results over conventional QA models, with 90% reporting enjoying this new media form.

In this work, we consider the problem of automatic content retrieval (ACR) using joint audio-video fingerprints. We focus on how to balance the query accuracy and the size of fingerprint, and how to allocate the fingerprint bits to video and audio frames to maximize the query accuracy. By introducing a novel concept called coverage, which is highly correlated to the query accuracy, we are able to construct a rate-coverage model and formulate the joint audio-video fingerprint bit rate allocation into a dynamic programming optimization problem. Our experimental results demonstrate that, compared to existing approaches, our method improves the retrieval accuracy by up to 25% while using 60% of the original fingerprint bit rate.

In this paper, we develop a new approach of spatially supervised recurrent convolutional neural networks for visual object tracking. Our recurrent convolutional network exploits the history of locations as well as the distinctive visual features learned by the deep neural networks. Inspired by recent bounding box regression methods for object detection, we study the regression capability of Long Short-Term Memory (LSTM) in the temporal domain, and propose to concatenate high-level visual features produced by convolutional networks with region information. In contrast to existing deep learning based trackers that use binary classification for region candidates, we use regression for direct prediction of the tracking locations both at the convolutional layer and at the recurrent unit. Our experimental results on challenging benchmark video tracking datasets show that our tracker is competitive with state-of-the-art approaches while maintaining low computational cost.

In this paper, we propose an iterative method to address the face identification problem with block occlusions. Our approach utilizes a robust representation based on two characteristics in order to model contiguous errors (e.g., block occlusion) effectively. The first fits to the errors a distribution described by a tailored loss function. The second describes the error image as having a specific structure (resulting in low-rank in comparison with image size). We will show that this joint characterization is effective for describing errors with spatial continuity. Our approach is computationally efficient due to the utilization of the alternating direction method of multipliers. A special case of our fast iterative algorithm leads to the robust representation method, which is normally used to handle non-contiguous errors (e.g., pixel corruption). Extensive results on representative face databases (in constrained and unconstrained environments) document the effectiveness of our method over existing robust representation methods with respect to both identification rates and computational time.

In this work we present a general framework for robust error estimation in face recognition. The proposed formulation allows the simultaneous use of various loss functions for modeling the residual in face images, which usually follows non-standard distributions, depending on the image capturing conditions. Our method extends the current vast literature offering flexibility in the selection of the residual modeling characteristics but, at the same time, considering many existing algorithms as special cases. As such, it proves robust for a range of error inducing factors, such as, varying illumination, occlusion, pixel corruption, disguise or their combinations. Extensive simulations document the superiority of selecting multiple models for representing the noise term in face recognition problems, allowing the algorithm to achieve near-optimal performance in most of the tested face databases. Finally, the multi-model residual representation offers useful insights into understanding how different noise types affect face recognition rates.

Video retrieval and video copy detection are well studied problems. The goal is to find the matching video in a database from a given query video. Typically, these query videos are short and aligning the query video is of secondary importance. Short sequences can be aligned using dynamic time warping. But, since time and memory usage increases quadratically with the length of the sequences, such process is not suitable for the alignment of two full length movies. A typical feature film is between 70 and 210 minutes long. Our goal is to find an accurate frame-by-frame alignment of a full length original film and a copy that has inserted and deleted sequences (e.g., commercial breaks or censorship), as well as differences in quality, format and framerate. We propose a fast, robust and memory efficient video sequence alignment algorithm which has linear space and time complexity.

Smartphones and tablets with their apps pervaded our everyday life, leading to a new demand for search tools to help users find the right apps to satisfy their immediate needs. While there are a few commercial mobile app search engines available, the new task of mobile app retrieval has not yet been rigorously studied. Indeed, there does not yet exist a test collection for quantitatively evaluating this new retrieval task. In this paper, we first study the effectiveness of the state-of-the-art retrieval models for the app retrieval task using a new app retrieval test data we created. We then propose and study a novel approach that generates a new representation for each app. Our key idea is to leverage user reviews to find out important features of apps and bridge vocabulary gap between app developers and users. Specifically, we jointly model app descriptions and user reviews using topic model in order to generate app representations while excluding noise in reviews. Experiment results indicate that the proposed approach is effective and outperforms the state-of-the-art retrieval models for app retrieval.

The goal of this special issue is to provide a premier forum for researchers to present their recent research results on multimedia big data. It follows the recent success event—the First IEEE International Conference on Multimedia Big Data (BigMM 2015) that took place at the Chinese National Convention Center in Beijing, China, from April 20–22, 2015 It also provides an important opportunity for multidisciplinary work connecting big data to multimedia computing.

Smart TVs have realized the convergence of TV, Internet , and PC technologies, but still do not provide a seamless content interaction for TV-enabled shopping. To purchase interesting items displayed in a TV show, consumers must resort to a store or the Web, which is an inconvenient way of purchasing products. The fundamental challenge in realizing such a use case consists of understanding the multimedia content being streamed. Such a challenge can be realized by utilizing object detection to facilitate content understanding though it has to be executed as a computationally bound process so that consumers are provided with a responsive and exciting user interface. To this end, we propose a computational- and temporal-aware multimedia abstraction framework that facilitates the efficient execution of object detection tasks. Given computational and temporal rate constraints, the proposed framework selects the optimal video frames that best represent the video content and allows the execution of the object detection task as a computationally bound process. In this sense, the framework is computationally scalable as it can adapt to the given constraints and generate optimal abstraction results accordingly. Additionally, the framework utilizes “object views” as the basis for the frame selection process, which depict salient information and are represented as regions of interest (ROI). In general, an ROI can be a whole frame or a region that discards background information. Experimental results demonstrate the computational scalability of the proposed framework and the benefits of using the regions of interest as the basis of the abstraction process.

Adaptive wireless video scheduling has been widely studied to improve network performance. However, the majority of existing scheduling algorithms assume that they are able to converge instantaneously to adapt to a dynamic network state, that is, the execution time of the scheduling can be ignored. Nevertheless, due to the limited computation capacity of wireless nodes, this assumption is very difficult, sometimes even impossible, to satisfy in practice. This motivates us to address in this paper the following challenging question: what is the effect of the execution time on the scheduling performance? To this end, we first characterize the scheduling as a stochastic optimization problem that enables us to open up a new degree of performance to exploit in a tractable manner. Next, we build a connection between the execution time and video quality, and rigorously prove that the execution time is disadvantageous to the stability region, but advantageous to the flow balance. Therefore, these results are helpful to shed insights on fundamental scheduling guidelines on designing an efficient video transmission system.

This paper presents a binocular PTU (pan-tilt unit) camera video object tracking scheme using the MeanShift algorithm and the runtime disparity estimation. The proposed method is to accommodate the requirement of 3D content generation and accurate tracking in more advanced video surveillance applications. The disparity estimation process for each stereoscopic pair is formulated as an energy minimization problem. The iterative solution procedure is implemented in a course-to-fine manner. The estimated disparity is used to scale the tracking window by the MeanShift algorithm, i.e. the size of the tracking area is adjustable according to its inner disparity, and thus the moving object can be better located by the camera. The program maintains the semi-real-time performance and acceptable accuracy as evaluated on a set of standard test data. In our experiment, two PointGrey cameras are controlled through a PTU device. The disparity estimation process on the recorded tracking video (640×480) achieves 6fps on an ordinary PC (2.66GHz CPU, 4GB RAM).

In this paper we propose a novel framework for object retrieval based on automatic foreground object extraction and multi-layer information integration. Specifically, user interested objects are firstly detected from unconstrained videos via a multimodal cues method, then an automatic object extraction algorithm based on Grab Cut is applied to separate foreground object from background. The object-level information is enhanced during the feature extraction layer by assigning different weights to foreground and background pixels respectively, and the spatial color and texture information is integrated during the similarity calculation layer. Experimental results on both benchmark data set and real-world data set demonstrate the effectiveness of the proposed framework.

Most content-based recommender systems focus on analyzing the textual information of items. For items with images, the images can be treated as another information modality. In this paper, an effective method called MSLIM is proposed to integrate multimodal information for content-based item recommendation. It formalizes the probelm into a regularized optimization problem in the least-squares sense and the coordinate gradient descent is applied to solve the problem. The aggregation coefficients of the items are learned in an unsupervised manner during this process, based on which the k-nearest neighbor (k-NN) algorithm is used to generate the top-N recommendations of each item by finding its k nearest neighbors. A framework of using MSLIM for item recommendation is proposed accordingly. The experimental results on a self-collected handbag dataset show that MSLIM outperforms the selected comparison methods and show how the model parameters affect the final recommendation results.

Most video recommender systems limit the content to the metadata associated with the videos, which could lead to poor results since metadata is not always available or correct. Meanwhile, the visual information of videos is typically not fully explored, which is especially important for recommending new items with limited metadata information. In this paper, a novel content-based video recommendation framework, called Video Topic, that utilizes a topic model is proposed. It decomposes the recommendation process into video representation and recommendation generation. It aims to capture user interests in videos by using a topic model to represent the videos, and then generates recommendations by finding those videos that most fit to the topic distribution of the user interests. Experimental results on the Movie Lens dataset validate the effectiveness of Video Topic by evaluating each of its components and the whole framework.

Recent advances in mobile media cloud (MMC) make it possible for users to enjoy the multimedia applications at anytime and anywhere. Most existing scheduling algorithms for MMC assume that the system parameters, such as user demand rate and server service time, are known to the scheduler. However, this assumption is invalid in many practical scenarios. In this paper, we consider a blind scenario where the above system parameters are unavailable. We aim at developing a blind scheduling algorithm (BSA) that performs well across magnitudes of fairness, simplicity and asymptotic optimality for a relatively general MMC. Specifically, the blind scheduling is first formulated as a finite time horizon optimization problem and fairness is required to be maintained at each time point with a given probability from the scheduler. Next, BSA routes the new users to the media service provider (MSP) whose weighted idle time is the longest, then assigns the available MSPs according to the fairness on the idle time. Importantly, we demonstrate that BSA is asymptotically optimal in the Halfin-Whitt heavy traffic (HWHT) regime. The asymptotic optimality is in the sense that the scheduling asymptotically and stochastically minimizes the steady-state waiting time of all the users. Our analysis also shows that in the HWHT regime the heterogeneous MSP system outperforms its homogeneous MSP counterpart in terms of the user waiting time. Moreover, we apply BSA to the program recommendation system and investigate its property.

Most existing Quality of Experience (QoE)-driven multimedia resource allocation methods assume that the QoE model of each user is known to the controller before the start of the multimedia playout. However, this assumption may be invalid in many practical scenarios. In this paper, we address the resource allocation problem with incomplete information where the realized mean opinion score (MOS) can only be observed over time, but the underlying QoE model and playout time are unknown. We consider two variants of this problem: 1) the form of the QoE model is known but the parameters are unknown; 2) both the form and the parameters of the QoE model are unknown. For both cases, we develop dynamic resource allocation schemes based on online test-optimization strategy. Simply speaking, one first spends appropriate time on testing the QoE model, then optimizes the sum of the MOS in the remaining playout time. The highlight of this paper lies in resolving the inherent tension between the test and optimization by jointly considering the uncertainties of QoE model and playout time. Furthermore, we derive tight bounds on the MOS loss incurred by the proposed schemes in comparison with the optimal scheme that knows the QoE model a priori and prove that the performance gap, as the playout time tends to infinity, asymptotically shrinks to zero.

This paper introduced a camera surveillance system in wireless communications. The system contains three major modules, PTU (pan-tilt unit) camera control for surveillance video capture, cross-layer control for data compression and transmission, and error concealment for video quality enhancement. Our contribution is twofold. First, a system design for data collection and transmission over wireless networks is presented and is evaluated with physical surveillance equipments. The camera is capable of following the moving target according to the control information. The end-to-end distortion estimation in the delay constrained video coding process takes into account the dynamic channel condition and physical layer modulation and coding scheme (MCS) to determine optimal coding and transmission parameters. Second, multiple error concealment strategies, including interleaving, boundary match and video up-sampling, are applied utilizing the special property of the PTU camera motion.

In this work, we investigate the impact of mobility on video streaming over multi-hop wireless networks by utilizing a class of scheduling schemes. We show that node spatial mobility has the ability to improve video quality and reduce the transmission delay without the help of advanced video coding techniques. To describe a practical mobile scenario, we consider a random walk mobility model in which each node can randomly and independently choose its mobility direction, and all the nodes can identically and uniformly visit the entire network. The contributions of this work are twofold: 1) It studies the optimal node velocity for the mobile video system. In this case, it is possible to achieve almost constant transmission delay and video quality as the number of nodes increases; 2) It derives an achievable quality-delay tradeoff range for different node velocities. Therefore, it is helpful to shed insights on network design and fundamental guidelines on establishing an efficient mobile video transmission system.

In this panel, each panelist will present their view of the current state-of-the-art of research and product innovations in the three major areas of multimedia experience: visual, auditory and gaming. We will discuss examples of innovation that enhance the consumption and sharing of multimedia (video, audio, graphics etc.) and thus increase quality of user experience. Another major focus of this panel is to open the discussion on how to innovate new multimedia user experiences.

Most existing works on routing for video transmission over multihop wireless networks only focus on how to satisfy the network-oriented quality-of-service (QoS), such as through put, delay, and packet loss rate rather than application-oriented QoS such as the user-perceived video quality. Although there are some research efforts which use application-centric video quality as the routing metric, they either calculate the video quality based on some predefined rate-distortion function or model without considering the impact of video coding and decoding (including error concealment) on routing, or use exhaustive search or heuristic methods to find the optimal path, leading to high computational complexity and/or suboptimal solutions. In this paper, we propose an application-centric routing framework for real-time video transmission in multihop wireless networks, where expected video distortion is adopted as the routing metric. The major contributions of this paper are: 1) the development of an efficient routing algorithm with the routing metric expressed in terms of the expected video distortion and being calculated on-the-fly, and 2) the development of a quality-driven cross-layer optimization framework to enhance the flexibility and robustness of routing by the joint optimization of routing path selection and video coding, thereby maximizing the user-perceived video quality under a given video playback delay constraint. Both theoretical and experimental results demonstrate that the proposed quality-driven application-centric routing approach can achieve a superior performance over existing network-centric routing approaches.

Routing for video transmissions over multi-hop wireless networks has gained increasing research interest in recent years. However, most existing works only focus on how to satisfy the network-oriented QoS, such as, throughput, delay, and packet loss rate rather than the user perceived quality. Although there are some research efforts which use application-centric video quality as the routing metric, the calculation of video quality is based on some predefined rate-distortion function or model without exploring the impact of video coding and decoding (including error concealment) on network path selection and the resulting received video quality. Moreover, unlike network- centric routing metrics, such as, hop count, average delay or average success probability of packet transmission, video distortion cannot be calculated either additively or multiplicatively in a hop-by-hop fashion due to the dependency among packets introduced by error concealment. As a result, most existing works use either exhaustive search or heuristic methods to find the optimal path, which leads to high computational complexity or suboptimal solutions to the routing problem of video transmission. In this paper, we propose an application- centric routing framework for real-time video transmission over multi-hop wireless networks, where expected video distortion is used as the routing metric. The major contributions of this work are: 1) the development of an efficient routing algorithm with the routing metric in terms of the expected video distortion being calculated on-the-fly, and 2) the development of a quality-driven cross-layer optimization framework to enhance the flexibility and robustness of routing by the joint optimization of routing path selection and video coding, thereby maximizing the user perceived video quality under a given video playback delay constraint. Both theoretical and experimental results demonstrate that the proposed quality-driven application-centric routing approach can achieve a superior performance over existing network-centric routing approaches.

The 14 papers in this special issue are divided into four categories: theoretical frameworks and schemes; integration of media processing and network protocols; heterogeneity; and quality evaluation.

TCP Friendly Rate Control (TFRC) has been designed to provide smoother sending rate than TCP for multimedia applications. However, most existing work on TFRC is restricted within exploring the performance of TFRC itself in wired or wireless networks without considering the interaction between TFRC and other network layers. This paper proposes a quality- driven TFRC framework for real-time video streaming, where real-time video coding at the application layer and the packet sending rate at the transport layer are jointly optimized. The proposed framework is formulated to find the optimal video coding parameters and the sending rate to minimize the end- to-end expected video distortion under a given video playback delay constraint. The proposed framework has been implemented and tested by using both H.264/AVC codec and NS-2 simulator. Experimental results demonstrate that the proposed joint optimization framework can significantly improve the received video quality over the existing schemes, especially when delay bound is tight.

Video transport over multi-hop wireless networks has received significant research interests recently. The majority of the research efforts in this field have been conducted taking the approach of cross-layer optimization. However, video content and user perceived quality have been largely ignored in existing work. In this paper, we integrate video content analysis into video transport over wireless mesh networks (WMN). A content-aware quality-driven cross-layer optimization framework is proposed to achieve the best end-to-end user perceived video quality. In our framework, the extracted video regions of interest (ROI) are discriminatingly coded, transmitted and protected in video encoding, network routing and packet scheduling by different network layers. We aim at the optimization of key parameters of each layer while focusing on their interactions across the holistic network protocol stack. The proposed framework is evaluated by H.264/AVC codec and WMN simulations. Experimental results demonstrate that the proposed framework can effectively provide a good user perceived video quality, especially when the delay requirement is stringent.

In this paper we propose a cross-layer optimized framework which jointly considers the video coding and transmission in wireless mesh networks. In our framework, video coding parameters, network path selection, MAC layer frame size, and modulation and coding schemes at PHY layer are systematically optimized across the entire network protocol stack. To achieve the goal of real-time video communication with high-quality and/or stringent delay requirements, the proposed framework is formulated into a minimum distortion and/or minimum delay problem. The formulated problem is efficiently solved as a Lagrange dual problem, and the solution is tested by H.264/AVC. The system performance in terms of the perceived video quality is compared between the cross-layer optimized system and the current system. Experimental results indicate the effectiveness and efficiency of the proposed optimization framework.

This paper presents a new multi-mode video object segmentation scheme to automatically segment head-and-shoulder objects from a video sequence. The proposed system supports two modes of segmentation: intra-mode and inter-mode. The intra-mode segmentation integrates skin detection, facial feature localization and verification, object shape approximation, split-and-merge region growing, and object region selection and guarantees good performance of the proposed segmentation scheme. The inter-mode object segmentation makes use of background modeling and subtraction to take advantage of temporal correlation of video frame while increasing the robustness of the segmentation and speeding up the performance. The main contributions of the proposed method includes: (i) a robust and efficient background modeling in inter-mode object segmentation, (ii) a facial feature verification and a multi-face separation algorithm in intra-mode segmentation. The performance of the algorithm is illustrated by simulation results carried out on various head-and-shoulder video sequences.

Video summarization has gained increased popularity in the emerging multimedia communication applications, however, very limited work has been conducted to address the transmission problem of video summary frames. In this paper, we propose a cross-layer optimization framework for delivering video summaries over wireless networks. Within a rate-distortion theoretical framework, the source coding, allowable retransmission, and adaptive modulation and coding have been jointly optimized, which reflects the joint selection of parameters at physical, data link and application layers. The goal is to achieve the best video quality and content coverage of the received summary frames and to meet the delay constraint. The problem is solved using Lagirangian relaxation and dynamic programming. Experimental results indicate the effectiveness and efficiency of the proposed optimization framework, especially when the delay budget imposed by the upper layer applications is small, where more than 10% distortion gain can be achieved.

In this paper, we propose a low-complexity content-adaptive background skipping scheme for region-of-interest (ROI) video coding in real-time wireless video telephony applications. Based on the real-time content information of the current and previous frames, such as foreground shape deformation, foreground and background motion, and background texture complexity, the proposed algorithm dynamically decides whether to skip Non-ROI macroblocks of current frame and reallocate saved bits to ROI and coded non-ROI. In this work, we proposed and compared three bit reallocation strategies, and the one predicting the future events using a Bayesian model is found the most efficient. Experimental results indicate that the proposed scheme significantly outperforms other methods by up to 2.5dB.

This work presents a low-complexity system implementation of a novel content-adaptive background skipping scheme for region-of-interest (ROI) video coding in mobile video phone applications. To improve the overall perceptual quality, the proposed approach reallocates bits to ROI macroblocks by adaptively skipping the non-ROI and using a weighted bit allocation scheme. The skip decision for the non-ROI is adaptively determined by the content information of the video, such as foreground shape deformation, foreground motion, background motion and background texture complexity. Experimental results demonstrate that the proposed scheme outperforms traditional schemes by up to 2 dB.

Video summarization has gained increased popularity in the emerging multimedia and communications applications. However, very limited work has been conducted to address the transmission problem of video summary frames. In this paper, we propose a cross-layer optimization framework for delivering video summaries over wireless networks. Within a rate-distortion theoretical framework, the source coding, allowable retransmission, and adaptive modulation and coding have been jointly optimized, which reflects the joint selection of parameters in physical, data link and application layers. The goal here is to achieve the best video quality and content coverage from the received summary frames as well as meet the delay constraint. In this paper, this problem is formulated and solved by using Lagrangian relaxation and dynamic programming. Simulation results indicate the effectiveness and efficiency of the proposed optimization framework, especially when the delay budget imposed by the upper layer applications is small, where more than 12% distortion gain can be achieved

In recent years, joint source-channel coding for multimedia communications has gained increased popularity. However, very limited work has been conducted to address the problem of joint source-channel coding for object-based video. In this paper, we propose a data hiding scheme that improves the error resilience of object-based video by adaptively embedding the shape and motion information into the texture data. Within a rate-distortion theoretical framework, the source coding, channel coding, data embedding, and decoder error concealment are jointly optimized based on knowledge of the transmission channel conditions. Our goal is to achieve the best video quality as expressed by the minimum total expected distortion. The optimization problem is solved using Lagrangian relaxation and dynamic programming. The performance of the proposed scheme is tested using simulations of a Rayleigh-fading wireless channel, and the algorithm is implemented based on the MPEG-4 verification model. Experimental results indicate that the proposed hybrid source-channel coding scheme significantly outperforms methods without data hiding or unequal error protection.

In object-based video encoding, the encoding of the video data is decoupled into the encoding of shape, motion, and texture information, which enables certain functionalities, like content-based interactivity and content-based scalability. The fundamental problem, however, of how to jointly encode this separate information to reach the best coding efficiency has not been studied thoroughly. In this paper, we present an operational rate-distortion optimal scheme for the allocation of bits among shape, motion, and texture in object-based video encoding. Our approach is based on Lagrangian relaxation and dynamic programming. We implement our algorithm on the MPEG-4 video verification model, although it is applicable to any object-based video encoding scheme. The performance is accessed utilizing a proposed metric that jointly captures the distortion due to the encoding of the shape and texture. Experimental results demonstrate that the gains of lossy shape encoding depend on the percentage the shape bits occupy out of the total bit budget. This gain may be small or may be realized at very low bit rates for certain typical scenes.

Object-based video coding is a relatively new technique to meet the fast growing demand for interactive multimedia applications. Compared with conventional frame-based video coding, it consists of two types of source data: shape information and texture information. Recently, joint source-channel coding for multimedia communications has gained increased popularity. However, very limited work has been conducted to address the problem of joint source-channel coding for object-based video. In this paper, we propose a cost-distortion optimal unequal error protection (UEP) scheme for object-based video communications. Our goal is to achieve the best video quality (minimum total expected distortion) with constraints on transmission cost and delay in a lossy network environment. The problem is solved using Lagrangian relaxation and dynamic programming. The performance of the proposed scheme is tested using simulations of a narrow-band block-fading wireless channel with additive white Gaussian noise and a simplified differentiated services Internet channel. Experimental results indicate that the proposed UEP scheme can significantly outperform equal error protection methods.

The joint source-network encoding of object-based video is a very important and challenging research topic, which has not been adequately explored. In this paper, we propose a robust network-adaptive encoding approach for object-based video. The framework jointly considers source coding, packet loss during transmission, and error concealment at the decoding. The proposed method guarantees the minimum expected distortion for the decoded video, by optimally allocating the shape and texture coding parameters at the encoder. The resulting optimization problem is solved by Lagrangian relaxation and dynamic programming. Experimental results demonstrate that the proposed method has significant gains over the non network-adaptive method.

We study the joint source-channel coding of object-based video, and propose a data hiding scheme that improves the video error resilience by adaptively embedding the shape and motion information in the texture data. Within a rate-distortion theoretical framework, the source coding, channel coding, data embedding, and decoder error concealment are jointly optimized based on the knowledge of transmission channel conditions. The problem is solved using Lagrangian relaxation and dynamic programming. Experimental results indicate that the proposed hybrid source-channel coding scheme significantly outperforms methods without data hiding or unequal error protection.

In this paper, we consider dynamic resource allocation for object-based wireless video communications. In object-based video coding, a video frame is comprised of objects that are described by their shape as well as their texture. By jointly considering source coding, error concealment and transmission power management at the physical layer, the proposed framework minimize the expect distortion at the receiver for given energy and delay constraints. In order to provide unequal error protection for the shape and texture information, a new video packetization scheme is proposed. Experimental results indicate that the proposed unequal error protection schemes significantly outperform equal error protection methods.

In this paper, we propose an optimal unequal error protection scheme for object-based video communications over differentiated services networks. Our goal is to achieve the best video quality (minimum total expected distortion) with constraints on transmission cost and delay. An end-to-end distortion estimation approach for object-based video is proposed, which can be used for different packetization schemes. The problem is solved using Lagrangian relaxation and dynamic programming. Experimental results indicate that the proposed unequal error protection schemes can significantly outperform equal error protection methods.

In object-based video, the encoding of the video data is decoupled into the encoding of shape, motion and texture information, which enables certain functionalities like content-based interactivity and scalability. However, the problem of how to jointly encode these separate signals to reach the best coding efficiency has never been solved thoroughly. In this paper, we present an operational rate-distortion optimal bit allocation scheme that provides a solution to this problem. Our approach is based on the Lagrangian relaxation and dynamic programming. Experimental results indicate that the proposed optimal encoding approach has considerable gains over an ad-hoc method without optimization. Furthermore the proposed algorithm is much more efficient than exhaustive search.

In this paper, we consider the rate-distortion optimal encoding of shape information using a skeleton decomposition and the minimum maximum (minmax) distortion criterion. For bit budget constrained video communication applications, whose goal is to achieve as low as possible but almost constant distortion, the minmax criterion is the natural choice. We propose a 4D DAG (directed acyclic graph) shortest path algorithm implemented by dynamic programming to solve the minimum rate problem, and also provide a solution for the dual minimum distortion problem. Experimental results indicate that our algorithm has an outstanding performance compared with existing methods.

MPEG-4 is the first multimedia standard that supports the decoupling of a video object into object shape and object texture information, which consequently brings up the optimal encoding problem for object-based video. In this paper, we present an operational rate-distortion optimal bit allocation scheme between shape and texture for MPEG-4 encoding. Our approach is based on the Lagrange multiplier method, while the adoption of dynamic programming techniques enables its higher efficiency over the exhaustive search algorithm. Our work will not only benefit the further study of joint shape and texture encoding, but also make possible the deeper study of optimal joint source-channel coding of object-based video.

In this paper, we present a new shape-coding approach, which decouples the shape information into two independent signal data sets; the skeleton and the boundary distance from the skeleton. The major benefit of this approach is that it allows for a more flexible tradeoff between approximation error and bit budget. Curves of arbitrary order can be utilized for approximating both the skeleton and distance signals. For a given bit budget for a video frame, we solve the problem of choosing the number and location of the control points for all skeleton and distance signals of all boundaries within a frame, so that the overall distortion is minimized. An operational rate-distortion (ORD) optimal approach using Lagrangian relaxation and a four-dimensional direct acyclic graph (DAG) shortest path algorithm is developed for solving the problem. To reduce the computational complexity from O(N(5)) to O(N(3)), where N is the number of admissible control points for a skeleton, a suboptimal greedy-trellis search algorithm is proposed and compared with the optimal algorithm. In addition, an even more efficient algorithm with computational complexity O(N(2)) that finds an ORD optimal solution using a relaxed distortion criterion is also proposed and compared with the optimal solution. Experimental results demonstrate that our proposed approaches outperform existing ORD optimal approaches, which do not follow the same decomposition of the source data.

We present a new shape-coding approach, which decouples the shape information into two independent data sets, the skeleton and the distance of the boundary from the skeleton. The major benefit of this approach is that it allows a more flexible trade-off between accuracy of the approximation and bit-allocation cost, and thus, provides the possibility of better performance in the operational rate-distortion (ORD) optimal sense than other reported techniques. The characteristics of these data sets are studied and various approximation approaches are applied on each of them to reach an ORD optimal result. We apply, for example, polygonal approximation on both the skeleton and distance data. We demonstrate that the resulting approach outperforms existing ORD optimal approaches.