With the mobile and portable devices become ubiquitous for people's daily life, how to design user interfaces of these products that enable natural, intuitive and fun interaction is one of the main challenges the multimedia community is facing. Following previous successful events, the third ACM International workshop on Interactive Multimedia on Mobile and Portable Devices (IMMPD'13) aims to bring together researchers from both academia and industry in domains including computer vision, audio and speech processing, machine learning, pattern recognition, communications, human-computer interaction, and media technology to share and discuss recent advances in interactive multimedia.

When mobile and portable devices become ubiquitous for people's daily life, how to design multimedia user interfaces of these products that enable natural, intuitive and fun interaction is one of the main challenges the multimedia community is facing. Following several successful events, the 2nd ACM International workshop on Interactive Multimedia on Mobile and Portable Devices (IMMPD'12) aims to bring together researchers from both academia and industry in domains including computer vision, audio and speech processing, machine learning, pattern recognition, communications, human-computer interaction, and media technology to share and discuss recent advances in interactive multimedia.

This paper shows that the use of Weighted Finite-State Transducer (WFST) significantly eliminates large-scale ambiguity in scene text recognition, especially for Japanese Kanji characters. The proposed method consists of two WFSTs called WFST-OCR and WFST-Lexicon. WFST-OCR handles the multiple hypotheses caused by erroneous text location, character segmentation and character recognition processes. The following WFST-Lexicon and its convolution of WFST-OCR resolve the hypotheses. The WFSTs integrate the conventional OCR and post-processing processes into one process. The benefit from the proposed method is that all the ambiguities are held as WFST data, and solved in one integrated step, the system outputs texts that are statistically consistent with regard to segmentation possibilities and the given language model. An experimental system demonstrates practical performance in spite of the hypothesis complexity inherent in the ICDAR test set and Kanji character texts.

With the mobile and portable devices become ubiquitous for people's daily life, how to design user interfaces of these products that enable natural, intuitive and fun interaction is one of the main challenges the multimedia community is facing. Following several successful events, the ACM International workshop on Interactive Multimedia on Mobile and Portable Devices (IMMPD'11) aims to bring together researchers from both academia and industry in domains including computer vision, audio and speech processing, machine learning, pattern recognition, communications, human-computer interaction, and media technology to share and discuss recent advances in interactive multimedia.

This panel paper presents motivations for discussing mobile media search and contains statements from the panelists who are industry research leaders in this field.

Recently, many exciting media search applications have been introduced to take advantage of smart phones' audiovisual capture capabilities and their being always on and connected. These applications address a real pain point for most mobile users and allow them to search with minimal text entry, if any. Is the mobile platform an ideal fit for media search? Are audio and visual signal processing technologies sufficiently accurate to support most mobile search applications? What are the killer applications of mobile media search? Earlier in 2009 at ICASSP, a panel on this topic stirred up great interest and enthusiasm while leaving many questions untouched due to the limited time.

The ACM International workshop on Interactive Multimedia for Consumer Electronics (IMCE) aims to bring together researchers from both academia and industry in domains including computer vision, machine learning, audio and speech processing, communications, artificial intelligence and media technology to share and discuss recent advances in interactive user interfaces and multimedia applications. Multimedia interaction is becoming a technology applied in many consumer electronics devices and can make user interfaces more intuitive and controllable. Multiple modalities including audio, video and haptics can be utilized and fused for media interaction.

In this paper, we address the problem of evaluating perceptual visual quality of compressed video under different settings for mobile communication. A 5-dimensional video feature space is constructed with the codec type, video content, frame size, frame rate and bitrate. The video quality assessment is formulated as a transformation between the multidimensional feature space and the quality space. Subject viewing tests for videos coded with H.263 and H.264 at different bitrates and spatial/temporal resolutions are performed. Based on statistical analysis, we find that the perceptual quality of the decoded video is affected by the encoder type, video content, bitrate, frame rate and frame size, in a descending order for significance. The methodology and findings in this paper designate new researching direction in video quality assessment and can be extended to other applications.

Most studies in the literature for video quality assessment have been focused on the evaluation of quantized video sequences at fixed and high spatial and temporal resolutions. Only limited work has been reported for assessing video quality under different spatial and temporal resolutions. In this paper, we consider a wider scope of video quality assessment in the sense of considering multiple dimensions. In particular, we address the problem of evaluating perceptual visual quality of low bit-rate videos under different settings and requirements. Extensive subjective view tests for assessing the perceptual quality of low bit-rate videos have been conducted, which cover 150 test scenarios and include five distinctive dimensions: encoder type, video content, bit rate, frame size, and frame rate. Based on the obtained subjective testing results, we perform thorough statistical analysis to study the influence of different dimensions on the perceptual quality and some interesting observations are pointed out. We believe such a study brings new knowledge into the topic of cross-dimensional video quality assessment and it has immediate applications in perceptual video adaptation for scalable video over mobile networks.

We propose a frequency-domain independent component analysis (ICA) with robust and computationally-light post processing method for background noise reduction in mobile phone speech communication. In our scenario, multi-source signal separation is not the target, but noise reduction is the primal one. This primal target characterizes our approach that promotes a new physical constraint, in other words, we place a restriction on the amplitude range of the transfer functions rather than assuming that the amplitudes are constant. When there are diffraction, obstacles and reflections in the real-world environment, it is better to assume that transfer function amplitude (derived from the distance to the mouth) varies within a certain range. Our two-microphone experiment shows that the ICA-based noise reduction significantly improves speech recognition performance especially in severe noise conditions.

In the last five years, mobile network applications have been characterized by the following multimedia applications: e-mail, web browsing, games, video-clip & music download, and multimedia mail. Those are early applications, in the emerging 3G mobile network era, ported from "fixed-line" Internet. Thus, those mobile applications have been considered as a degraded version of Internet applications due to limitation of available bandwidth, latency, connection reliability and I/O device capability, where cellphones are also considered as miniature portable PCs. Borrowing many ideas from Internet, however, the mobile infrastructure has brought a highly-personalized popular communication culture that was never seen before.

This paper reviews practical video delivery technologies, examining existing mobile networks, commercialized or standardized transport, and coding technologies. Compression efficiency, power dissipation, and error control are intrinsic issues in wireless video delivery. Among these issues, error control technologies are evaluated at four layers: 1) layer-1/2 transport; 2) end-to-end transport layer such as TCP/IP or RTP/UDP/IP; 3) error-resilience tool and network adaptation layer) and 4) source coder layer. Layer-1/2 transport tends to provide two distinct conditions: one quasi-error-free, in which upper layer error control technologies show a limited improvement, and one with a burst of errors during the fading period, in which the adaptability of error control is essential. Emerging mobile network quality of service will have a large variation of bandwidth and delay. Thus, adaptive rate control and error recovery are identified as more crucial issues for future research.

In this paper we present a video coder based on a redundant dictionary that employs nonseparable bidimensional functions. The algorithm used for the representation of the residual of motion estimation is matching pursuit (MP). The proposed dictionary has new features capable of catching curved and oriented contours typical of the residual of motion estimation in natural video sequences. An adaptive grid approach is adopted for the coding of atoms and this is combined with an innovative coding method employing an arithmetic encoder. The proposed coder shows improvements over H.264 in terms of compression efficiency, with up to 20% gains for very low bitrates.

It is known that the human visual system (HVS) does not pay equal attention to each error and even region in judging picture quality. In this paper, we combine a perceptual model with an integrated detection of the spatially transitional regions in visual distortion evaluation to better match the HVS perception to visual quality. For decompressed images or video, the spatially transitional regions are the regions where major perceptually disturbing artefacts caused by edge impairments (mainly due to blurring and locations where the edge information is not adequately represented) and the presence of false edges (mainly due to blockiness and the presence of strong rippling effects of ringing) usually occur. Such regions are efficiently detected based on a single two-dimensional spatial high-pass filter in our work. Good correlation between the proposed method and the human perception has been demonstrated with the full set of 50-Hz video quality expert group test data.

We describe a structured variable length coding (VLC) based on Golomb code that generalize the current context-based adaptive variable length coding (CAVLC) in the emerging video coding standard H.264/AVC. The current CAVLC is characterized as unstructured VLC and uses extensive dedicated code tables. Lack of generality due to these code tables causes a problem of over-fitting or overlearning when we estimate a set of coding parameters. We propose a "compact" structured code based on Golomb code providing generality, extensibility, and low implementation complexity with only three parameters. Introduction of a simple truncation method enables to avoid reversed order code length and remove unused code space for finite set of symbols. Experimental results show its similar coding efficiency to current CAVLC while reducing the size of memories to store code tables and providing adaptability for various probability functions.

Environmentally-adaptive XML transformation is a key goal for content adaptation. It needs a standard and computationally efficient framework for transformation engine that can transform input XML data depending on environment descriptions (e.g. terminal capabilities) and rule descriptions (e.g. access policies). We propose a method to realize the framework. We adopt XSLT and DOM processors: the DOM processors use environment descriptions and rule descriptions in composing XLST instances; the XLST processors refer to the XLST instances in transforming the input XML data. We cache the XLST instances for reuse. Since one XLST instance is produced for each environment description, their number is manageable; XLST instances are not necessarily created for every input. Caching the XLST instances and reducing the number of XLST instances created yields a significant computational advantage in large-scale multimedia content delivery. We will use the proposed framework to create an AAA (authentication, authorization, and accounting) system for multimedia content delivery and network control.

We propose a scalable security description framework for mobile Web services that takes account of service context such as time constraints, purchase prices, and network connection status. We also present a secure open interface for multi-level authorization procedures. To describe AAA service flows, the framework adopts the Web service flow language (WSFL) and the Web service description language (WSDL); it uses the security assertion markup language (SAML) to describe security certificates. We extend WSDL so that it offers security attributes that specify the underlying secure protocols. Combining WSFL, WSDL, SAML, and service context descriptions yields scalable AAA services that will support mobile e-commerce by realizing high-speed and effective micropayments.

Matching pursuit (MP) based video coder represents an active alternative to conventional block based DCT based architectures R Neff et al. (1997). In this paper we propose a generalized motion compensation method and an entropy coding method for MP based coder. Motion vectors are combined with the cut-off frequency level of loop filters applied to the reference frame. Exploiting the characteristics of MP atoms, the latter are efficiently coded by employing an adaptive grid size combined with context based adaptive arithmetic coder. Reordering of atoms in each grid is also applied. Simulation results show that our proposed method outperforms traditional fixed grid based approach.

In this paper, we describe a low-overhead MB based motion model proposed to a coding candidate architecture of emerging ITU-T video coding standard H.26L and show that it outperforms the current H.26L motion model at a lower range (e.g., 16Kbps–64Kbps) of bitrate. The current H.26L's motion compensation employs a granularity scaling strategy, which scales accuracy of motion compensation depending on an available bitrate. This results in the degradation of the prediction performance in case of lower bitrate coding, where the bit budget for motion representation has to be reduced. We propose a better harmony of overhead and gain in prediction, by introducing a motion boundary model in place of H.26L's motion compensation, that provides a better prediction performance at the lower bitrates. Our experimental results show that the proposed model achieves bit-rate saving up to 5% compared with H.26L. A Rate-Distortion analysis verifies the advantage of the proposed motion model.

In this paper, two new algorithms have been proposed for performing video object segmentation. In both algorithms, a new method has been proposed for adaptive threshold estimation for the change detection process. The method is able to provide a reliable estimation of the threshold without knowing explicitly which data corresponds to the moving and non-moving elements in the video sequence. In addition, the use of new specially detected “moving” edgels and also a graph-based edge-linking method that employs both gradient-based and edge-based constraints have also been proposed in order to refine the object segmentation mask. The second algorithm further incorporates an initialisation phase to solve the problems of dis-occluded regions appearing as part of the segmented moving objects. The proposed work has shown a reasonably fast and good video object segmentation technique that operates in real-time and can be applied in applications such as MPEG-4's content-based video coding and home/office/warehouse security. Comparison with one of the MPEG-4 adopted video object segmentation techniques shows that our approach is very significantly faster and yet produces better results.

We propose a of control architecture for mobile multimedia streaming in which RTP monitoring agents report QoS information to media servers. The RTP monitoring agents lie midway between wired networks and radio links, monitor RTP packets sent from media servers to mobile terminals, and report quality information to media servers so that the servers can realize network-adaptive QoS control. By analyzing the information sent from the agents, the servers can distinguish quality degradation caused by network congestion from that caused by radio link errors, and can improve service quality by controlling the transmission rate and robustness against packet loss. Simulation results show our implemented QoS control schemes that use the feedbacks from the agent and control transmission rate and robustness provide better video quality than a simple rate control mechanism.

We propose a motion compensation (MC) scheme that outperforms the MC technique specified in the emerging ITU-T coding standard, H.26L. The proposed MC scheme is based on a motion boundary model that uses two different time/frame references for arbitrarily bisected regions within each macroblock. The motion boundary model increases MC efficiency especially for the incoherent moving regions caused by the intersection between foreground and background objects; the current H.26L's "motion boundary insensitive MC", on the other hand, allows only one reference frame for all segments within a macroblock to be predicted. Experiments show that the proposed MC scheme can reduce the bit rate by up to 5.9% compared to the current H.26L, without increase of decoding complexity.

This paper presents a method for distortion-optimized streaming of predictively coded video over packet networks with varying delay. In networks with significant delay variations, coded video frames can arrive late at the decoder and miss their respective display deadlines. Furthermore, due to predictive coding, a late frame can also prevent a number of subsequent frames from being displayed properly, where the number of affected frames or degree of distortion depends on the particular coding dependencies of the late frame. In this paper, we present an optimized video streaming strategy based on frame reordering for networks with significant delay variations. This streaming strategy minimizes distortion by exploiting the fact that different late frames result in different degrees of distortion. We model the router-induced delay in a wired network with an analytical PDF and we model the link-layer retransmission delay of a wireless network with the 3GPP specification for W-CDMA radio link control. We compute the distortion for different frame reorderings using the network delay models and a source model that accounts for the prediction dependencies of predictively coded video. Our optimized streaming strategies are shown to reduce the number of late frames by 14 to 23% for the situations examined.

In this paper, we present a mobile streaming media CDN (Content Delivery Network) architecture in which content segmentation, request routing, pre-fetch scheduling, and session handoff are controlled by SMIL (Synchronized Multimedia Integrated Language) modification. In this architecture, mobile clients simply follow modified SMIL files downloaded from a streaming portal server; these modifications enable multimedia content to be delivered to the mobile clients from the best surrogates in the CDN. The key components of this architecture are 1) content segmentation with SMIL modification, 2) on-demand rewriting of URLs in SMIL, 3) pre-fetch scheduling based on timing information derived from SMIL, 4) SMIL updates by SOAP (Simple Object Access Protocol) messaging for session handoffs due to clients mobility. We also introduce QoS control with a network called an RTP monitoring agent to enable appropriate control of media quality based on both network congestion and radio link conditions. The current status of our prototyping on a mobile QoS testbed MOBIQ is reported in this paper. We are currently designing the SOAP-based APIs (Application Programmable Interfaces) needed for the mobile streaming media CDN and building the CDN over the current testbed.

This paper proposes a closed-loop optimization framework to improve image coding efficiency by searching DCT coefficients at the equivalent subjective quality to the original coding result. The proposed framework shares a basic idea currently adopted in speech coding that searches optimal codes in a closed-loop operation, evaluating the coded signal with perceptually weighted mean square error. To evaluate the perceptual quality in image coding, we introduce the masked PSNR that accounts for the masking effects, by which we apply the stepwise removal of subjectively negligible DCT coefficients. The result justifies the effectiveness of the proposed framework.

NTT DoCoMo is the company to "DO Communications over the Mobile Network", and has 36 million subscribers in April 2001. This presentation covers DoCoMo's multimedia services over IMT-2000. DoCoMo's IMT-2000 network (i.e., one of Japan's third generation mobile networks) allows us to have transmission speeds of 384 kbps. With the benefit of this broader bandwidth, mobile multimedia services are expected to run on top of the network. Mobile visual phones and media streaming on demand are typical applications, as well as enhanced "i-mode" services that provide us with the ability to access Internet contents written in compact HTML, and to exchange e-mails faster than ever. For such applications, we have newly developed coding technologies such as MPEG-4, and related system technologies. After introducing the applications for wireless video with its coding technologies, future challenges and directions beyond the IMT-2000 network are discussed.

A macroblock-based optimization method is described for selecting quantization parameters so as to reduce the re-quantization errors in MPEG transcoding. Analysis has shown that re-quantization using a parameter larger than the original input quantization parameter but smaller than twice the original causes significant information loss in transcoding. This problem can be avoided by using a parameter larger than twice the original. The optimization method was thus extended based on a rate-distortion model to enable the re-quantization parameter to be selected from ones larger than twice the original input quantization range. This improved method outperformed a typical non-optimized transcoding scheme, improving the PSNR by about 1 dB.

We propose a gray-scale character recognition method for video indexing. It is robust against the problems of binarization against a complex background and low resolution. Unlike a traditional character recognition scheme through image binarization, we directly extract Gabor features (called Gabor jets) from video contents. The use of the Gabor filters contributes to freeing a tricky binarization process for cluttered images, and furthermore provides localized directional edge features, which have phase-shift invariance to edge positions. To form a feature vector to be classified, we accumulate the extracted Gabor features along projection lines in local regions, and then categorize them with a standard LVQ classifier. The projective accumulation provides robustness under character deformation caused by variation of font types or imprecise segmentation. We compare the proposed method by experiments with a typical OCR method, for which correct binarization is advantageously given. The proposed method shows similar or superior performance to the other method in understanding video captions.

We present a formulation for estimation of structure and motion parameters for a roaming robot that has translation on the floor and rotation along the vertical axis. To estimate the parameters, we use at least five feature correspondences among three viewpoints, where we ignore those vertical positioning. This is because the vertical changes in pointwise matching between two views is generally subtle in roaming robot environment, then it is practically hard for a traditional structure-from-motion estimation method (i.e., a pointwise matching method). Focusing on the horizontal changes, we use "directional feature" in panoramic images; we formulate the problem as the estimation of depth of the directional features and location of the viewpoints in the reference coordinate. Owing to the proposed method, more than 1000 panoramic images have been registered with calibrated location, and used for image-based rendering in computer graphics.

This paper presents image-based rendering related techniques: (1) a computer vision technique to estimate accurate epipolar geometry and (2) an image coding technique to compress multiple-viewpoint images, which are stored for reconstruction of any arbitrary view of 3-D object, with the epipolar geometry. Unlike other epipolar geometry estimation techniques based on the correct sets of feature point correspondence, which is hardly given in practice, the proposed method minimizes absolute luminance difference between two images with respect to seven epipolar geometry parameters. In the framework of block matching, the absolute luminance difference between the two images can be represented by the seven parameters with which a nonlinear robust regression follows. We apply the epipolar geometry estimation technique to multiple-viewpoint image coding. That practically contributes to improving the coding efficiency, where more than 1000 viewpoint images are coded by a type of predictive coding based on epipolar geometry. The experimental results show that the compression ratio attains to 1/100 and more.

Block matching is a basic tool in motion estimation, and widely used in image coding. In this paper, we give a parametric representation to block matching, called "parametrized block correlation". The parametrized block correlation is a quadratic scalar function that represents a local luminance difference associated with the local displacement. By further associating the local displacement with 2D parametric motion models, we can obtain the overall motion parameters that minimize the sum of parametrized block correlations. The advantages in using the parametrized block correlation are justified through the experimental results.

We describe an image coding scheme based on multiple templates for interactive audio-visual (AN) database retrieval. The image sequence to be coded consists of overlapping image planes with opacity information and depth ordering. In this method, each image plane is independently encoded to a different bit stream where each video object sequence is reconstructed from representative frames (i.e., templates) by global and local deformations. Owing to this coding scheme, the following new functionalities are supported: identification and manipulation of two-dimensional (2-D) video objects, selective decoding and browsing of visual contents, and very high compression efficiency. We have extended an MPEG1 coding software to an object-based coding software. Experimental results of the proposed scheme prove the above advantages.

A central problem in estimating multiple image motions is that 2-D motion estimation and region segmentation are mutually dependent. The authors present a new region description method for dealing with this mutual dependence problem. Segmentation and motion estimation are simultaneously performed by a clustering process based on color, motion, and pixel position. As a result of the clustering, an image is decomposed into region fragments. Each fragment is characterized by distribution parameters of color, pixel positions, and spatiotemporal intensity gradients. The image is described by the parameters of the region fragments; 2-D motion vectors for each fragment are obtained from the distribution parameters of the intensity gradients. By combining those features in the clustering process, regions are segmented more precisely, motion boundaries are not blurred, and the 2-D motions are obtained even in noisy areas. Experimental results are presented.>