We present a topology-informed inverse rendering approach for reconstructing high-genus surface meshes from multi-view images. Compared to 3D representations like voxels and point clouds, mesh-based representations are preferred as they enable the application of differential geometry theory and are optimized for modern graphics pipelines. However, existing inverse rendering methods often fail catastrophically on high-genus surfaces, leading to the loss of key topological features, and tend to oversmooth low-genus surfaces, resulting in the loss of surface details. This failure stems from their overreliance on Adambased optimizers, which can lead to vanishing and exploding gradients. To overcome these challenges, we introduce an adaptive V-cycle remeshing scheme in conjunction with a re-parametrized Adam optimizer to enhance topological and geometric awareness. By periodically coarsening and refining the deforming mesh, our method informs mesh vertices of their current topology and geometry before optimization, mitigating gradient issues while preserving essential topological features. Additionally, we enforce topological consistency by constructing topological primitives with genus numbers that match those of ground truth using Gauss-Bonnet theorem. Experimental results demonstrate that our inverse rendering approach outperforms the current state-of-the-art method, achieving significant improvements in Chamfer Distance and Volume IoU, particularly for high-genus surfaces, while also enhancing surface details for low-genus surfaces.

Neural representations for 3D meshes are emerging as an effective solution for compact storage and efficient processing. Existing methods often rely on neural overfitting, where a coarse mesh is stored and progressively refined through multiple decoder networks. While this can restore high-quality surfaces, it is computationally expensive due to successive decoding passes and the irregular structure of mesh data. In contrast, images have a regular structure that enables powerful super-resolution and restoration frameworks, but applying these advantages to meshes is difficult because their irregular connectivity demands complex encoder–decoder architectures. Our key insight is that a geometry image–based representation transforms irregular meshes into a regular image grid, making efficient image-based neural processing directly applicable. Building on this idea, we introduce our neural geometry image–based representation, which is decoder-free, storage-efficient, and naturally suited for neural processing. It stores a low-resolution geometry-image mipmap of the surface, from which high-quality meshes are restored in a single forward pass. To construct geometry images, we leverage Optimal Transport (OT), which resolves oversampling in flat regions and undersampling in feature-rich regions, and enables continuous levels of detail (LoD) through geometry-image mipmapping. Experimental results demonstrate state-of-the-art storage efficiency and restoration accuracy, measured by compression ratio (CR), Chamfer distance (CD), and Hausdorff distance (HD).

Animating 3D head meshes using audio inputs has significant applications in AR/VR, gaming, and entertainment through 3D avatars. However, bridging the modality gap between speech signals and facial dynamics remains a challenge, often resulting in incorrect lip syncing and unnatural facial movements. To address this, we propose OT-Talk, the first approach to leverage optimal transportation to optimize the learning model in talking head animation. Building on existing learning frameworks, we utilize a pre-trained Hubert model to extract audio features and a transformer model to process temporal sequences. Unlike previous methods that focus solely on vertex coordinates or displacements, we introduce Chebyshev Graph Convolution to extract geometric features from triangulated meshes. To measure mesh dissimilarities, we go beyond traditional mesh reconstruction errors and velocity differences between adjacent frames. Instead, we represent meshes as probability measures and approximate their surfaces. This allows us to leverage the sliced Wasserstein distance for modeling mesh variations. This approach facilitates the learning of smooth and accurate facial motions, resulting in coherent and natural facial animations. Our experiments on two public audio-mesh datasets demonstrate that our method outperforms state-of-the-art techniques both quantitatively and qualitatively in terms of mesh reconstruction accuracy and temporal alignment. In addition, we conducted a user perception study with 20 volunteers to further assess the effectiveness of our approach.

In this work, we present a method to address the problem of novel view and time synthesis of complex dynamic scenes considering the input video is subject to blurriness caused due to camera or object motion or out-of-focus blur. Neural Scene Flow Field (NSFF) has shown remarkable results by training a dynamic NeRF to capture motion in the scene, but this method is not robust to unstable camera handling which can lead to blurred renderings. We propose Deblur-NSFF, a method that learns spatially-varying blur kernels to simulate the blurring process and gradually learns a sharp time-conditioned NeRF representation. We describe how to optimize our representation for sharp space-time view synthesis. Given blurry input frames, we perform both quantitative and qualitative comparison with state-of-the-art methods on modified NVIDIA Dynamic Scene dataset. We also compare our method with Deblur-NeRF, a method that has been designed to handle blur in static scenes. The demonstrated results show that our method outperforms prior work.

SSD (Single Shot Detector) is one of the state-of-the-art object detection algorithms, and it combines high detection accuracy with real-time speed. However, it is widely recognized that SSD is less accurate in detecting small objects compared to large objects, because it ignores the context from outside the proposal boxes. In this paper, we present CSSD-a shorthand for context-aware single-shot multibox object detector. CSSD is built on top of SSD, with additional layers modeling multi-scale contexts. We describe two variants of CSSD, which differ in their context layers, using dilated convolution layers (DiCSSD) and deconvolution layers (DeCSSD) respectively. The experimental results show that the multi-scale context modeling significantly improves the detection accuracy. In addition, we study the relationship between effective receptive fields (ERFs) and the theoretical receptive fields (TRFs), particularly on a VGGNet. The empirical results further strengthen our conclusion that SSD coupled with context layers achieves better detection results especially for small objects (+3.2%[email protected] on MSCOCO compared to the newest SSD), while maintaining comparable runtime performance.

Recent years have witnessed the emergence and ongoing proliferation of dynamic adaptive streaming over HTTP (DASH1), which reuses web servers with HTTP communication instead of relying on RTSP/RTP/RTCP-based media server and promises to be capable of automatically tuning to bandwidth dynamics. Aware of its excellent performance, the third generation partnership project (3GPP) long term evolution (LTE) has adopted DASH (with specific codecs and operating modes) for use over mobile wireless networks in order to realize ubiquitous multimedia delivery. In a multi-user multiple-input-multiple-output (MU-MIMO) LTE system, spatial multiplexing gain can be achieved by making sure the transmitter to deliver distinct data streams to multiple receivers simultaneously, which provides the choices to opportunistically schedule the preferred receivers each time for a common time-frequency resource. In such a system, one of the major challenges to enhance DASH performance is to design an effective scheduler that can fully enjoy the benefit of spatial reuse as well as guaranteeing satisfactory video services for all users. To this end, in this paper, we propose a utility maximization framework (UMF) for DASH application delivered over MU-MIMO LTE downlinks. In particular, we characterize DASH performance by a combined utility function in terms of average video rate, playback buffer status, and battery energy state. Correspondingly, we develop a utility-based scheduler that selects multiple user equipments (UEs) to share each common network resource under the consideration of precoding-based MU-MIMO links in order to maximize system-wide DASH performance. We prove the NP-hardness of the scheduling problem and propose a priority search algorithm to provide time-efficient solution. We further incorporate novel rate adaptation on the application layer for the scheduled UEs to dynamically set the requested encoding bitrates to explore the balance between agile responsiveness and shifting smoothness. Extensive system-level simulations with realistic video trace validate the effectiveness of our framework in terms of rate adaptability, playback buffer depletion percentage, and battery energy saving.

Image stitching has been developed to enhance users' immersive experiences by constructing new images of higher spatial resolution and broader field of view from multiple input images. Most image stitching schemes assume 2D transformation of the whole frame between images. There usually exist local misalignments of contents that do not match with the transformation and may cause artifacts such as ghosting. Several schemes including seam cutting, image blending and other image composition algorithms, have been developed to relieve such artifacts. However, these schemes are essentially post-stitching processing that do not address the alignment problem and would fail when the parallax is large. In this paper, we propose a multi-objective content preserving warping (CPW) for local adjustment of image alignment which is optimized considering feature matching, line preserving, line matching, shape smoothness and other factors to significantly reduce the visual artifacts. In particular, we adjust the image warping from their pre-warping results using the homography estimated during the image registration step. Experimental results show that the proposed CPW scheme for image stitching is able to generate much smoother stitching results comparing with the state-of-the-art methods reported in the literature.

Video stitching requires proper seam cutting technique to decide the boundary of the sub video volume cropped from source videos. In theory, approaches such as 3D graph-cuts that search the entire spatiotemporal volume for a cutting surface should provide the best results. However, given the tremendous data size of the camera array video source, the 3D graph-cuts algorithm is extremely resource-demanding and impractical. In this paper, we propose a sequential seam cutting scheme, which is a dynamic programming algorithm that scans the source videos frame-by-frame, updates the pixels' spatiotemporal constraints, and gradually builds the cutting surface in low space complexity. The proposed scheme features flexible seam finding conditions based on temporal and spatial coherence as well as salience. Experimental results show that by relaxing the seam continuity constraint, the proposed video stitching can better handle abrupt motions or sharp edges in the source, reduce stitching artifacts, and render enhanced visual quality.

Providing well-satisfactory multi-party video conferencing service is remarkably challenging, which demands not only high bandwidth but also low latency. Current commercial implementations typically use one or multiple multipoint control units (MCUs) as the central points for distributing video bit-streams to all participants in the conferencing session. Such MCU-based solution has limited control on quality of service (QoS), which may cause large delay, single-point malfunction and communication bottleneck for entire system. Recent years have witnessed the emergence of a new paradigm in networking, software defined networking (SDN), which advocates separating data plane and control plane, making network switches in the data plane simple packet forwarding devices and leaving a logically centralized controller to manipulate network behaviors. SDN provides the flexibility of changing underlying infrastructure and makes it possible to radically provide service/flow-aware QoS guarantee. In this paper, we propose a novel architecture for multi-party video conferencing by utilizing SDN-enabled multicasting, where SDN controller helps media controller to buildup multicast trees for video flows originated at video parties. To realize the architecture, we correspondingly propose a novel multicast construction and packing method, in which multiple source-based multicast trees are constructed and integrated to maximize system-wide utility while guaranteeing an end-to-end delay bound. Extensive simulations demonstrate that it could provide better video delivery compared to the conventional MCU-based solution in terms of video rate and delay in both sparsely and densely distributed networks.

Long scene panorama (LSP) can provide the immersive experience for an extended view in one single photograph. The state-of-the-art image mosaicing based LSP techniques are significantly constrained by their high computational complexity, limiting their applications to platforms with rich computing resource. Slit-scanning based schemes, although implementation friendly, heavily rely on the consistency of the camera's speed and suffer from pushbroom distortion. In this paper, we propose a novel scheme which can generate the LSP for the indoor environment via consumer-grade cameras. A novel adaptive resampling scheme is introduced to generate more natural looking panoramas from videos captured by cameras with non-constant moving speed. A multiple slits replacement scheme is also developed to reduce the zigzag distortion and to handle the depth change. Experimental results show that the proposed scheme can achieve satisfactory performance with handheld cameras.

Most works about affective image classification in computer vision treat each emotion category independently and predict hard labels, ignoring the correlation between emotion categories. In this work, inspired by psychological theories, we adopt a dimensional emotion model to model the correlation among certain emotion categories. We also propose a framework of changing image emotion by using our emotion predictor. Easily extendable to other feature transformations, our framework changes image emotion by color histogram specification, relaxing the limitation of the previous method that each emotion is associated with a monotonic palette. Effective and comparable to the previous work of changing image emotion shown by user study, our proposed framework provides users with more flexible control in changing image emotion compared with the previous work.

In recent years, the performance of image denoising has been boosted drastically by nonlocal algorithms and sparse coding techniques. In this paper, we also take a nonlocal approach to image denoising and formulate the problem as one of collaborative LMMSE estimation from grouped image patches. We show that our optimal LMMSE solution amounts to shrinking the singular values of the matrix representation of the grouped image patches. This interpretation of our solution allows us to relate our estimation-theoretic approach to other nonlocal algorithms and sparse coding techniques in the literature. In addition, we develop an iterative algorithm to find the best LMMSE estimate. Experimental results show that our proposed denoising algorithm achieves better PSNR and subjective performance than the state of the art.

In many applications, users desire to access a random viewing angle of a scene in high resolution while this specifically queried imagery is not among the acquired sample images. In theory, 3D reconstruction based rendering can generate such an image. However, accurate camera calibration over large scale photo collections is needed and is highly complex in nature. Image stitching based approachescan also be applied. However, such scheme is unable to provide free view interpolation or resolution enhancement. In this paper, we present a novel free view image super-resolution scheme to interpolate free views for planar structures. We construct a Bayesian model and marginalize it over photometric regulation and geometric registration parameters applying the Lie group theory. Experimental results show that the proposed scheme is able to achieve desired performance against the state-of-the-art image super-resolution approaches and successfully obtain registration in full 6 degree-of-freedom (DOF). Compared with existing image based rendering schemes, the proposed scheme achieves free view interpolation for planar structures with higher resolution and less distortion.

Hand-held mobile phone photography usually suffers from motion blur. Without information about the camera motion, it is difficult for traditional methods to remove such blur. We study sensor-assisted single image deblurring on modern smartphones. Information about the camera motion is obtained from the smartphones' built-in sensors such as ac-celerometers, gyroscopes, and magnetometers. Addressing the characteristics of mobile photo capture, we propose a camera model to estimate the point spread function (PSF) based on the 3D camera orientation computed from the fused sensor data. A convenient framework is developed for automatically calibrating the camera and sensors using a series of consecutively captured photos, without requiring any extra device. In addition, to accommodate users' sensitivity in artifacts over human faces, a face-adaptive deblurring framework is proposed. Features measuring image quality are computed over the automatically detected face regions, based on which the appropriate deblurring algorithm is selected. We evaluate our approach over 500 consumer photos featuring various levels of blur and illumination, where our algorithm shows clear advantages compared with state-of-the-art traditional blind and non-blind deconvolution methods in both de-blurring quality and speed.

We study real-time one-pass video stabilization on mobile devices such as smartphones and tablets. A localized camera path planning framework is proposed, where through maintaining a forward or backward-looking buffer window, the short-term camera trajectory is locally optimized according to cinematographic rules. Also, we propose a hybrid auto-corrective path optimization algorithm, where the system automatically switches among different motion models according to the actual video adaptively. Using a multithreading mechanism, the proposed method can be implemented on mobile devices to achieve real-time one-pass video stabilization. Evaluation on a large variety of consumer videos demonstrates the effectiveness of our system.

Recent years have witnessed the emergence and ongoing proliferation of dynamic adaptive streaming over HTTP (DASH1), which reuses web servers with HTTP communication instead of relying on RTP-based media server and promises to be capable of automatically tuning to bandwidth dynamics. The third generation partnership project (3GPP) long term evolution (LTE) has adopted DASH for use in order to realize ubiquitous multimedia delivery. In a multi-user multiple-input-multiple-output (MU-MIMO) LTE system, spatial multiplexing gain can be achieved by making the transmitter deliver distinct data streams to multiple receivers simultaneously, which provides choices to schedule preferred receivers for a common resource. In such a system, one of the major challenges to enhance adaptive HTTP streaming performance is to design an effective scheduler that can fully enjoy the benefit of spatial reuse as well as guaranteeing satisfactory video services for all users. To this end, we propose a utility maximization framework (UMF) for DASH application over MU-MIMO LTE downlinks. In particular, we characterize DASH performance by a combined utility function in terms of average video rate, playback buffer status, and battery energy state. Correspondingly, we develop a utility-based scheduler that selects multiple user equipments (UEs) to share each common network resource under the consideration of precoding-based MU-MIMO in order to maximize system-wide DASH performance. Specifically, we propose a priority search algorithm to provide time-efficient solution to the scheduling problem. We further incorporate novel rate adaptation on the application layer for scheduled UEs to dynamically set the requested video encoding bitrates. Extensive system-level simulations validate the effectiveness of UMF in terms of rate adaptability, playback buffer depletion percentage and battery energy consumption.

We present in this paper a new device-to-device (D2D) communication scheme to create a QoS enhanced multimedia services for LTE users who are physically close to each other. This scheme has a potential to provide higher bandwidth service with reduced delay and reduced power consumption. However, a significant challenge from such D2D opportunity is to establish the direct link between nearby devices without interference to other regular LTE users. In this research, we develop a novel user selective resource allocation scheme that allows D2D links to share the same radio resources with LTE regular users while proactively avoiding the interferences with intelligent user selection. The main contribution of the proposed scheme is in the innovative design of the sub-channel selection algorithms for both D2D links and regular LTE users. These algorithms serve dual purpose: (1) providing desired QoS enhancement to multimedia consumers in D2D links while minimizing the interference to regular users; (2) maximizing the sum rate of the LTE networks considering the interference from D2D links and the fairness issue. The simulation results show that the total achievable capacity of LTE networks is dramatically enhanced by D2D communication with this new user selective resource allocation.

Data redundancy elimination (DRE), also known as data de-duplication, reduces the data amount to be transferred or stored by identifying and eliminating both intra-object and inter-object duplicated data elements. It is one of the key content delivery acceleration techniques over wide area networks (WANs) to reduce delivery latency and bandwidth consumptions by reducing the amount of data to be transferred. Deploying DRE at the end hosts maximizes the bandwidth savings and latency reductions, because the amount of content sent to the destination hosts is minimized. However, standard DRE used to identify redundant content chunks is very expensive in terms of memory and processing capability especially on resource constrained hosts. By analyzing the web application traffic traces, we find out that some types of contents have more redundant contents than others. Thus, it is possible to apply DRE selectively and opportunistically on those contents with more redundant data elements than other content types to save the memory and processing resources at the hosts. In this paper, we propose content-type based selective DRE (SDRE), which deploys DRE selectively on the contents which have the most opportunities for redundant content identification. We explore the benefits of deploying SDRE on smartphone traffic traces. The results show that SDRE can achieve almost the same bandwidth savings as that of standard DRE with less computation and smaller memory.

Owing to the imminent fixed mobile convergence, Internet applications are frequently accessed through mobile nodes. However, service delivery latency is too high to satisfy user expectations. In this paper, we design a new TCP algorithm, TCP-ME (Mobile Edge), to accelerate the service delivery in mobile networks. Considering the QoS (Quality of Service) mechanisms of mobile networks, TCP-ME is designed to differentiate the packet loss caused by wireless errors, traffic conditioning of mobile core networks, and Internet congestion, as well as to react to the packet loss accordingly. To detect wireless errors, we mark the ACK (Acknowledge) packets in the uplink direction at the base station, and the marking threshold is a function of the instantaneous downlink queue length and the number of consecutive HARQ retransmissions. We modify the ECN mechanism with deterministic marking to detect Internet congestion. The packet loss caused by traffic conditioners of mobile networks is detected by whether the incoming DUPACK is marked or not. TCP-ME adapts the inter-packet interval when the packet loss is caused by wireless errors or the admission control mechanism. If the packet loss is due to Internet congestion, TCP-ME applies the TCP-New Reno's congestion window adaptation algorithm. Simulation results show that TCP-ME can speed up web service response time in mobile networks by about 80%.

Transmission Control Protocol (TCP) slow start degrades TCP performance under conditions of long-distance and high end-to-end latency, i.e., inherent characteristics of wide area networks (WANs). In this paper, we propose a new TCP slow start algorithm for WANs, called Adaptive Fast Start (AFStart), which incorporates an inline available bandwidth measurement over TCP technique into TCP slow start to set the slow start threshold adaptively and adjusts the congestion window intelligently. The performance of AFStart is evaluated through simulations using the dumb-bell topology and parking-lot topology by applying AFStart to Fast TCP. The simulation results show that AFStart can ramp up the congestion window from its initial value to the slow start threshold more quickly and smoothly than standard slow start, and AFStart achieves higher network link utilization and TCP throughput during the slow start than Fast TCP.

Embedding watermark into H.264 bit-streams directly can simplify encoding procedure and reduce computational cost comparing with conventional encoder based watermarking algorithms. However, possible error propagation may occur due to the prediction mechanism of the H.264 codec and cause video quality degradation. Expensive computation is often needed to compensate such drift. Furthermore, compensation in compressed domain usually leads to significant increase in the size of the watermarked video. To address these problems, we have developed a novel paradigm that can minimize the error propagation due to the watermarking process with very low complexity, high fidelity, robustness and security. This new paradigm is based on careful analysis of the dependency relation among the macro-blocks. We first extract the macro-block dependencies from the H.264 prediction mechanism and cache them as a graph. We then select these macro-blocks that have the least impact to the overall sequence to embed the desired watermark. This new way of embedding shall intrinsically minimize the error propagation without post processing of drift compensation. Extensive simulations have been carried out to confirm that the proposed watermark embedding is indeed able to achieve the desired performance with low complexity, high fidelity, robustness and security.

Video denoising based on temporal or spatiotemporal filtering is highly effective but computationally expensive due to the requirement of motion estimation. Encoder-integrated denoising is an efficient framework that embeds the filtering process into the encoding pipeline so that motion estimation for denoising can be avoided. State-of-the-arts encoder-integrated methods use Least Minimum Mean Square Error (LMMSE) optimal filters to reduce noise based on additive noise models. However, in practice, the LMMSE filters could result in blurred edges and visual artifacts due to inaccurate image signal estimation caused by outlier input samples and non-adaptivity within macroblocks. This paper presents new recursive bilateral filters that can be easily integrated into video encoders to tackle the limitations of the LMMSE filters. The robustness of the bilateral filters results in substantially better objective and subjective quality with marginal computation cost increase.

Blocking artifact reduction or deblocking algorithm is an important component in modern block-based video encoding architecture and often used as post-processing procedures in many encoding/transcoding applications. Most of the existing video deblocking algorithms do not take into account Human Visual System(HVS) models and employ empirically designed filters, resulting in suboptimal perceptual image quality and the difficulty to adjust parameters for different application scenarios. This paper presents a new deblocking algorithm that automatically generates spatially variant adaptive filters by maximizing a regularized perceptual quality metric. Such optimization based approach leads to better human perceived image quality and much easier filter parameter adjustment.

For the video retargeting problem which adjusts video content into a smaller display device, it is not clear how to balance the three conflicting design objectives: 1) visual interestingness preservation; 2) temporal retargeting consistency; and 3) nondeformation. To understand their perceptual importance, we first identify that the latter two play a dominating role in making the retargeting results appealing. Then a statistical study on human response to the targeting scale is carried out, suggesting that the global preservation of contents pursued by most existing approaches is not necessary. Based on the newly prioritized objectives and the statistical findings, we design a video retargeting system which, as a refined homogeneous approach, addresses the temporal consistency issue holistically and is still capable of preserving high degree of visual interestingness. In particular, we propose a volume retargeting cost metric to jointly consider the retargeting objectives and formulate video retargeting as an optimization problem in graph representation. A dynamic programming solution is then given. In addition, we introduce a nonlinear fusion based attention model to measure the visual interestingness distribution. The experiment results from both image rendering and subjective tests indicate that our proposed attention modeling and video retargeting system outperform their conventional methods, respectively.

This paper is proposing a new framework of virtual ads insertion for Augmented Reality (AR) in a specific real scene, building facades of the street view videos. In this framework, specific region detection, camera orientation estimation, visual tracking, motion filtering and virtual-real blending are discussed. For the building façade, vanishing points are fully identified to detect a rectangular planar structure. Meanwhile, dynamic registration and color harmonization are designed to improve visual acuity of virtual ads insertion. Experimental results are given to show the good performance of the proposed framework.

Encryption of multimedia video data has been extensively studied. However, a pipelined method of compression followed by encryption is not always feasible and springs new challenges, namely the effect of visual degradation and the computational cost of the encryption process. Recently, joint encryption and compression techniques, which aim to encrypt the video as it is being compressed by the encoder, have gained popularity. This work aims to develop a lightweight yet effective multi-layer key stream based technique to encrypt video data by performing joint encryption and compression. Our joint encryptor-encoder performs almost as fast as a normal H.264 compression-only encoder. The multi-layer key stream we use for encryption is extracted from the underlying video structure itself, thus thwarting (piracy) key generators. It also facilitates efficient handing of packet loss in real-time transmission. Results show our method produces almost complete obscuration of the video, a consistent degradation for diverse scenes, and format compliant encrypted H.264 video.

In this panel, each panelist will present their view of the current state-of-the-art of research and product innovations in the three major areas of multimedia experience: visual, auditory and gaming. We will discuss examples of innovation that enhance the consumption and sharing of multimedia (video, audio, graphics etc.) and thus increase quality of user experience. Another major focus of this panel is to open the discussion on how to innovate new multimedia user experiences.

A compelling video summarization should allow viewers to understand the summary content and recover the original plot correctly. To this end, we materialize the abstract elements that are cognitively informative for viewers as concepts. They implicitly convey the semantic structure and are instantiated by semantically redundant instances. Then we analyze that a good summary should i) keep various concepts complete and balanced so as to give viewers comparable cognitive clues from a complete perspective ii) pursue the most saliency so that the rendered summary is attractive to human perception. We then formulate video summarization as an integer programming problem and give a ranking based solution. We also propose a novel method to discover the latent concepts by spectral clustering of bag-of-words features. Experiment results on human evaluation scores demonstrate that our summarization approach performs well in terms of the informativeness, enjoyability and scalibility.

Video retargeting (resolution adaptation) is a challenging problem for its highly subjective nature. In this paper, a nonlinear saliency fusing approach, that considers human perceptual characteristics for automatic video retargeting, is being proposed. First, we incorporate features from phase spectrum of quaternion Fourier Transform (PQFT) in spatial domain and global motion residual based on matched feature points by the Kanade-Lucas-Tomasi (KLT) tracker in temporal domain. In addition, under a cropping-and-scaling retargeting framework, we propose content-aware information loss metrics and a hierarchical search to find optimal cropping window parameters. Results show the success of our approach on detecting saliency regions and retargeting on images and videos.

Video retargeting is the task of fitting standard-sized video into arbitrary screen. A compelling retargeting attempts to preserve most visual information of original video as well as deliver a temporally consistent retargeted view. To handle long video sequences, we perform the task on a shot/subshot basis. For each frame, a crop pane is determined to optimally select a region of interest as the retargeted frame in two stages: i.e. minimizing visual information loss (intra-frame consideration) to yield source and destination crop pane parameters at boundary frames and minimizing visual information loss accumulation under the visual inertness (inter-frame consideration) constraints to search for a smooth transition of crop pane across interior frames. The second minimization process is remodeled as the shortest-path problem in graph theory and the parametric transition of crop panes is solved by dynamic programming. Experiments demonstrate our approach preserves salient regions of original video whilst offering eye-friendly visual consistency.

IPTV, enabled by aggregating new technologies in computing, networking, and storage, provides a new information access and entertainment platform in the world of digital evolution. From this perspective where information and entertainment are part of each other, we write this paper to address the issues of IPTV towards seamless infotainment. We focus our discussions to explore such IPTV potentials from three aspects: information access and distribution, improving quality of entertainment, and enabling seamless infotainment. In addition, relevant topics such as IPTV standards and future directions are also discussed.

As the next generation network gains increasing attention, IPTV has been recognized as an efficient way to provide key IP multimedia services. Compared to conventional TV services, IPTV is sensitive to the IP network characteristics such as packet loss, delay and jittering. In addition, the potential long delay in channel switching is another significant challenge for IPTV systems. In this paper, to improve the QoS (quality of service) and QoE (quality of experience), we propose a framework which includes (1) introducing server-side bit-rate adaptation to deal with bandwidth variation; (2) multicasting low bit-rate videos for preview of multi-channel programs and Picture-in-Picture (PiP); and (3) using a two-step P-frame switching mechanism to reduce the delay in channel switching. Our results from a real-world test-bed show that the proposed framework greatly improves the perceived video quality and user experience at the client side.

Recent years have seen an exponential growth in the use of home networks, from the Internet-enabled PC to network-enabled home appliances. Ordinary and everyday appliances used in the home will increasingly become integral components of these networks. They are required to join, leave, and self-configure in accordance with their dynamic environment. New platforms and applications are needed to mediate interactions between devices to overcome the inherent problems, limitations, and costs of bespoke solutions. While combining our disparate devices to meet future needs is challenging, it will also allow for better exploitation of networked devices to provide obvious benefits to the consumer. A number of approaches and technologies that attempt to redress this issue exist; however, they are not without their own disadvantages. This paper discusses these technologies and introduces some of the novel approaches that have been proposed to address the issue of device autoconfiguration using peer-to-peer technologies. In this way, disparate devices are able to achieve self-management without user interaction. A case study is also presented to illuminate these approaches and demonstrate the applicability of our approach.

This paper provides a timely review of influential work in the area of peer-to-peer (P2P) content search. We begin with a survey of text-based P2P search mechanisms and continue with an exposition of content-based approaches followed by a discussion of future directions.

Peer-to-peer networking technologies are gaining increasing popularity for file sharing and media streaming applications. To support these set of applications, structured peer-to-peer (P2P) overlay networks may be employed to provide an overlay substrate. In structured P2P overlay networks, peer nodes, content objects and messages are often identified using a set of well defined identifications (Ids.) Objects are stored on peer nodes based on a set of predefined rules. Messages are routed towards the destination node, for example, the root node of the message Id, after an average of h routing hops. In the absence of faults, a high probability of successful message forwarding, i.e., a message is delivered correctly to the destination node, can be achieved even when a large fraction of the peer nodes crash. However, most current overlay networks are not secure. When one or more malicious nodes are presented in the overlay, they can prevent correct message delivery throughout the overlay. This paper studies attacks aimed at preventing correct message delivery in structured peer-to-peer overlays. Several different routing algorithms are analyzed to understand means to improve messaging security without significant additional cost.

Presents the welcome message from the conference proceedings.

Presents the welcome message from the conference proceedings.

This paper provides a brief overview of recent progress of peer-to-peer (P2P) technologies for multimedia applications. We provide an overview of the technical challenges, creative solutions and results related to P2P file sharing and content search, and P2P media streaming

Authentication is a primary component for secure delivery of multimedia content in many consumer networking applications: to guarantee the integrity of the received data stream. Meanwhile, consumer networks increasingly used for multimedia information and commercial content delivery, are destined to be heterogeneous. To provide QoS, it is necessary to adapt the multimedia stream to the heterogeneous network channel conditions and device capabilities. This new adaptation requirement, together with the 'best effort' and 'error prone' characteristics of IP and wireless networks, and the 'real time playback' property of streaming media impose challenges in the design of the streaming media authentication schemes. Conventional message authentication schemes do not offer suitable scalability or loss resilient capability for this new set of applications. A suitable authentication scheme must allow the recipient to authenticate the received data faster than the packet receiving rate, with means to adapt to channel conditions and device capability, despite any losses or transmission errors. In this paper we study the design of loss resilient scalable media stream authentication and present a scheme that is scalable and loss resilient.

Today, multimedia delivery through wireless networks is rapidly gaining popularity. Many cellular phones are already equipped with an extra large full-color display and various multimedia applications and devices. It is challenging to transmit and access multimedia data through wireless communications channels due to device mobility, power constraints, bandwidth limitations, and an error prone environment. We investigate multimedia services in wireless networks. We propose an integrated service approach for authentication and error control (including error detection, recovery and concealment) that potentially reduces bandwidth requirement and power consumption. Authentication is used in secure wireless communications to ensure the integrity of a message, i.e., that it has not been tampered with and that it originated with the presumed transmitter. To combat transmission errors with the least additional bandwidth requirement, error concealment can be used to obtain a close approximation of the original signal at the receiver end. The efficiency and effectiveness of an error recovery technique rely on the system error detection capabilities. Traditionally, those problems are treated separately using different schemes. We investigate the problem at the application layer, and design a joint media authentication and error control scheme to reduce cost and save resources. Simulation results show that the proposed scheme is efficient and effective.

Consumer networks, increasingly used for multimedia information and commercial content delivery, are destined to be heterogeneous. To provide QoS, it is necessary to adapt the multimedia stream to the heterogeneous network channel conditions and device capabilities. Meanwhile, security is an important component to restrict unauthorized multimedia content access and distribution. This suggests the need for a new cryptography system implementations that can operate at different data rates, i.e., be scaled to various multimedia content, different network topology, changing bandwidth, and diverse receiver device capabilities. Content authentication is one important security tool for secure multimedia content communication. Conventional message authentication schemes do not offer suitable scalability for this new set of applications. In this paper, we study the design of scalable media data stream authentication and present a framework for multimedia authentication that supports various kinds of scalability.

Over the last three years, mobile handsets have evolved from voice based services to multimedia terminals that is gradually leading towards a new era of consumer entertainment, fostered by mobile communications and consumer networks. The need for mobile digital rights management solutions is thus intensified in order to safeguard mobile media content. Various types of content protection solutions have been proposed. We investigate the challenge and state-of-the-art development of mobile multimedia content protection, describe scalable security tools that can support various kinds of scalability to provide adequate QoS for heterogeneous consumer networks, and discuss the applications for mobile consumer multimedia communications.

Traditional cryptography systems treat every portion of a message (a video, an image, or an audio) equally and encrypt the entire message as a whole. As a result, those security systems often have only two states: access authorization and access denial. To facilitate multi-level access control with interoperability, scalable security mechanism is needed. Further, the availability of varying network bandwidth and diverse receiver device capabilities demand scalable and flexible approaches that are capable of adapting to changing network conditions as well as device capabilities. We describe a multimedia encryption scheme that supports scalability. One desirable feature of the proposed scheme is its simplicity and flexibility in supporting scalable content access control.

Encryption is one of the most important security tools for secure message communication. There is an increasing need for secure digital multimedia communication technologies to restrict unauthorized access. Prior to the availability of variable bit rate (VBR) communication services, systems typically treat every portion of a message equally and encrypt each and every bit of a message that is decrypted at the same rate as encrypted. Hence, encryption/decryption processes at both ends of the network were required to operate at the same rate. The advent of VBR multimedia communication services require new cryptography system implementations that can operate at different data rates, i.e., be scaled to different speed, different network bandwidths, and diverse receiver device capabilities. In this paper, we describe a progressive and scalable encryption scheme that supports scalability. One desirable feature of the proposed scheme is its simplicity and flexibility in supporting scalable multimedia communication and multimedia content access control.

Traditional cryptography systems treat every portion of a message (a video, an image, or an audio) equally and encrypt the entire message as a whole. As a result, those security systems often have only two states: access authorization and access denial. To facilitate multi-level access control with interoperability, scalable security mechanism is needed. Further, the availability of varying network bandwidth and diverse receiver device capabilities demand scalable and flexible approaches that are capable of adapting to changing network conditions as well as device capabilities. In this paper, we describe a multimedia encryption scheme that supports scalability. One desirable feature of the proposed scheme is its simplicity and flexibility in supporting scalable content access control.

This paper applies the solutions to the fundamental issues addressed in Part I to specific design problems of embedding data in image and video. We apply multilevel embedding to allow the amount of embedded information that can be reliably extracted to be adaptive with respect to the actual noise conditions. When extending the multilevel embedding to video, we propose strategies for handling uneven embedding capacity from region to region within a frame as well as from frame to frame. We also embed control information to facilitate the accurate extraction of the user data payload and to combat such distortions as frame jitter. The proposed algorithm can be used for a variety of applications such as copy control, access control, robust annotation, and content-based authentication.

A novel AAC watermarking scheme with low structural and computational complexity is proposed to achieve real-time AAC audio watermark embedding capability. In this scheme, an enhanced spread spectrum watermarking method that increases the robustness and information hiding capacity of the watermarking channel is developed and employed. Furthermore, this scheme improves the speed of the encoding process in that we directly use the quantized indices rather than the dequantized coefficients, thus no explicit dequantization and requantization are performed. Our embedded watermark is tested to be robust against AAC/MP3 transcoding at a rate of 10 bps. The scheme is suitable for speedy encoding and decoding of watermark with low cost hardware.

Video authentication is expected to prove the integrity of the content. Common authentication techniques do not allow any form of manipulation, thus making it impractical for some applications such as transcoding. In this paper, we shall propose an authentication scheme which can survive the bit-rate-conversion transcoding methods by using public, imperceptible semi-fragile watermark. Nowadays, videos are only available in compressed format in many applications. We will focus on the authentication of MPEG video. We will first provide a new classification of video tampering methods. Based on that, we shall propose our own authentication system. The proposed approach has a good locality corresponding to MPEG structure.

The problem of digital watermarking for media authentication has attracted a considerable amount of attention in the last several years. Most previous works have mainly concentrated on authenticating a single medium, such as visual, audio or text medium. We discuss the multimedia authentication problem. That is, we study the problem of efficient data hiding for media authentication when at least two forms of media are presented in a digital data stream. In particular, we address the problem of multimedia data hiding for authentication purposes. A multilayer data hiding structure is presented. In addition, its implementation for our active data hiding application is discussed in detail.

Error concealment plays an important role in combating transmission errors. Methods of error concealment that produce better quality are generally of higher complexity, thus making some of the more sophisticated algorithms not suitable for real-time applications or restricted to client devices with limited capability. We propose an approach to use data hiding to facilitate the error concealment at the decoder. A set of features are extracted at the encoder and embedded imperceptibly into the host media. If some part of the media data is damaged during the transmission, the embedded features can be extracted and used for recovery of lost data. The use of data hiding leads to reduced complexity at the decoder. Simulation shows that our approach has better image quality than some well-known conventional error concealment methods.

We study certain aspect of data hiding problem, not for the classical application of copyright protection, but instead for media data recovery. A medium data, for example, an image, is partitioned into N components. A content-associative signature of each component is generated and inserted imperceptibly into remote region of the image. When imperfect transmission of image causes loss of blocks, the embedded corresponding signature is extracted and used to recover the lost blocks of data. Compared to conventional approaches, this new method can provide the knowledge of the lost block content, in addition to neighborhood information, to the decoder at the receiver end. With the knowledge of the lost block content, better restoration results can be reached, especially when conventional neighborhood interpolation method is used together with the data hiding approach.

Modern communication, networking, and multimedia technologies make it possible to distribution image and video electronically. More and more digital images and videos are available to the average users. Consequently, more and more users start to access digital image and video by means of new ways, such as through the Internet. Meanwhile, businesses also start seeking new technologies to enable new services that can accommodate the new trend. Techniques that facilitate easy and quick browsing, retrieval, manipulation, and secure access, transaction and purchasing of multimedia content through new communication channels are thus needed in order to provide new services. In this paper we propose a fast image/video retrieval methodology for easy and quick browsing and searching large image and video databases. Retrieval is done directly on the compressed images and videos, which enables real time performance.

This paper presents a new methodology for wipe transition identification. Shot transition detection is an important technique for making videos easier to handle. Due to the wide variety, wipe transition appears to be the most difficult one to be detected among all types of shot transitions. We propose an approach that takes advantage of the production aspect of video. Each video frame is first decomposed into low-resolution and high-resolution components which are analyzed respectively and further recombined together to form a wipe transition detector. In our system, wavelet transformation is used for multi-resolution decomposition.

A new methodology of image and keyframe retrieval for image and video databases is presented. In contrast to previous approaches, which only support similarity retrieval and direct image manipulation on-line, our system performs tagging off-line using neural network algorithm and answers query on-line using only the tags. This new method is more appealing in meeting user's needs than only emphasizing what current technology can offer. Experiments on our CAETI (Computer Assisted Education & Training Initiative) IML (Internet Multimedia Library) show that this model gives high quality query results with fast on-line performance. This visual search system is available at http:/www.videolib.princeton.edu/test/retrieve.