Omnidirectional videos are becoming more and more popular as display devices, such as Head Mounted Displays (HMD), become more user friendly. One of the biggest challenges with such video content is the great amount of bandwidth required for delivery at a good quality. The key difference between omnidirectional and the traditional content streaming is that with the former, users watch only part of the streamed content. Therefore, the non-watched areas of a streamed omnidirectional video are delivered unnecessarily. To reduce the network bandwidth requirements, the delivery mechanism could make use of Viewport-Dependent Streaming (VDS), according to which high-quality (HQ) video is maintained only in the current viewport, while lower quality video is streamed for the remaining part. In this paper, we examine the streaming data wastage, i.e., the parts of HQ video not watched by a user, in VDS when users linearly move from a generic viewport to another one within the 360 degrees space. We used both simulated trajectories as well real trajectories from 20 users. We show how the user's HMD exploration speed and the DASH transport segment lengths impact the streaming data wastage, leading up to 78% of wasted HQ content and up to 40% of total stream bit rate wasted due to users' motion. We also show how the users' trajectory prediction can decrease the streaming data wastage making the system up to 25% more efficient. This paper gives very useful hints that could lead to VDS optimizations, and it introduces quantitative estimates on how the data wastage characteristics change with respect to the streaming parameters.

Modern smartphones have become the most popular means for recording videos. In fact, thanks to their portability, smartphones allow for recording anything and at any moment of our everyday life. One common occasion is represented by sport happenings, where people often record their favourite team or players. Automatic analysis of such videos is important for enabling applications such as automatic organization, browsing and summarization of the content. This paper proposes novel algorithms for the detection of salient events in videos recorded at basketball games. The novel approach consists of jointly analyzing visual data and magnetometer data. The magnetometer data provides information about the horizontal orientation of the camera. The proposed joint analysis allows for a reduced number of false positives and for a reduced computational complexity. The algorithms are tested on data captured during real basketball games. The experimental results clearly show the advantages of the proposed approach.

This paper describes a user evaluation study of automated creation of mobile video remixes in three different event contexts. The evaluation contributes to the design process of the Automatic Video Remixing System, deepening knowledge to wider usage context. The study was completed with 30 users in three different contexts: a sports event, a music concert and a doctoral dissertation. It was discovered that users are motivated to provide their material to the service when knowing they get an automatically created remix containing many capturers' content in return. Automatic video remixing was stated to ease the task of editing videos and to improve the quality of amateur videos. The study reveals requirements for pleasurable remix creation in different event contexts and details the user experience factors related to the content capturing, sharing, and viewing of captured content and the remixes. The results provide insights into media creation in small event-based groups.

In this work we propose methods that exploit context sensor data modalities for the task of detecting interesting events and extracting high-level contextual information about the recording activity in user generated videos. Indeed, most camera-enabled electronic devices contain various auxiliary sensors such as accelerometers, compasses, GPS receivers, etc. Data captured by these sensors during the media acquisition have already been used to limit camera degradations such as shake and also to provide some basic tagging information such as the location. However, exploiting the sensor-recordings modality for subsequent higher-level information extraction such as interesting events has been a subject of rather limited research, further constrained to specialized acquisition setups. In this work, we show how these sensor modalities allow inferring information (camera movements, content degradations) about each individual video recording. In addition, we consider a multi-camera scenario, where multiple user generated recordings of a common scene (e.g., music concerts) are available. For this kind of scenarios we jointly analyze these multiple video recordings and their associated sensor modalities in order to extract higher-level semantics of the recorded media: based on the orientation of cameras we identify the region of interest of the recorded scene, by exploiting correlation in the motion of different cameras we detect generic interesting events and estimate their relative position. Furthermore, by analyzing also the audio content captured by multiple users we detect more specific interesting events. We show that the proposed multimodal analysis methods perform well on various recordings obtained in real live music performances.

The recent proliferation of mobile video content has emphasized the need for applications such as automatic organization and automatic editing of videos. These applications could greatly benefit from domain knowledge about the content. However, extracting semantic information from mobile videos is a challenging task, due to their unconstrained nature. We extract domain knowledge about sport events recorded by multiple users, by classifying the sport type into soccer, American football, basketball, tennis, ice-hockey, or volleyball. We adopt a multi-user and multimodal approach, where each user simultaneously captures audio-visual content and auxiliary sensor data (from magnetometers and accelerometers). Firstly, each modality is separately analyzed; then, analysis results are fused for obtaining the sport type. The auxiliary sensor data is used for extracting more discriminative spatio-temporal visual features and efficient camera motion features. The contribution of each modality to the fusion process is adapted according to the quality of the input data. We performed extensive experiments on data collected at public sport events, showing the merits of using different combinations of modalities and fusion methods. The results indicate that analyzing multimodal and multi-user data, coupled with adaptive fusion, improves classification accuracies in most tested cases, up to 95.45%.

We present a robust multimodal approach for classifying the sport genre in videos recorded by mobile phone users at a sport event. In addition to traditional audio-visual content analysis tools, we propose to analyze auxiliary sensor data (electronic compass data and accelerometer data) captured simultaneously with the video recording. By means of machine learning techniques, we build models of visual appearance, camera motion (from auxiliary sensor data) and audio scene, which are used for classifying the data from each modality. The sport genre is obtained by fusing the information provided by the models. We propose to use the quality of each modality as an indication of its reliability. Extensive experiments were performed on real test data collected at public sport events. We provide comparisons on the use of different modality sets and fusion methods. Finally, we show how the proposed methods achieve robust classification even in the considered unconstrained scenarios.

Digital memorabilia, such as video remixes, can increase the value of attending music events. Remixes can be made using video clips recorded by attendees during the event; however, producing them is a laborious task. In this paper we study the prospects of an automatic video remixing and present the results of a study on users' perceptions and attitudes towards collaborative automatic mobile video production. The three findings are as follows: People assess automatic video remix memorabilia as fairly equal to amateur-made manual ones, even if the manually-created video remixes are better in overall quality; as a remixing actor, a computer can be perceived to be more trustworthy than a human remixer; and, the quality of the video remix and the publication forum of the remix outcome plays a significant role when people are deciding whether or not they need public acknowledgement for their contribution. We conclude by discussing the design implications for collaborative automatic mobile video production.

Recording and publishing mobile video clips from music concerts is popular. There is a high potential to increase the concert's perceived value when producing video remixes from individual video clips and using them socially. A digital production of a video remix is an interactive process between human and computer. However, it is not clear what the collaboration implications between human and computer are.

Nowadays most camera-enabled electronic devices contain various auxiliary sensors such as accelerometers, gyroscopes, compasses, GPS receivers, etc. These sensors are often used during the media acquisition to limit camera degradations such as shake and also to provide some basic tagging information such as the location used in geo-tagging. Surprisingly, exploiting the sensor-recordings modality for high-level event detection has been a subject of rather limited research, further constrained to highly specialized acquisition setups. In this work, we show how these sensor modalities, alone or in combination with content-based analysis, allow inferring information about the video content. In addition, we consider a multi-camera scenario, where multiple user generated recordings of a common scene (e.g., music concerts, public events) are available. In order to understand some higher-level semantics of the recorded media, we jointly analyze the individual video recordings and sensor measurements of the multiple users. The detected semantics include generic interesting events and some more specific events. The detection exploits correlations in the camera motion and in the audio content of multiple users. We show that the proposed multimodal analysis methods perform well on various recordings obtained in real live music performances.

The Mobile and Interactive Social Television (MIST) demo introduces the concept of watching TV/Video content with people of interest via the mobile in a virtual shared space. In this shared space, the participants can interact and watch content together with others. The participants can talk to each other, see each other and also send text messages while watching the content. The coupling of content consumption and interaction allows users to have a common shared context of the viewing experience which creates a social experience that is both participative and interactive. The user feedback reveals that there is a clear and positive benefit and added value to the viewing experience.

Smart phones are becoming more and more powerful. Services that were traditionally designed for a static environment can now be implemented into mobile devices. One such service is Interactive Social TV, which allows geographically dispersed people to meet in a virtual shared space and watch TV while being able to interact with each other. This paper presents two novel architectures of a Mobile and Interactive Social TV system. In both of the architectures, the interaction is represented by a rich audio-visual media, allowing users to hear and see each other. In the first architecture, the mixing of the TV content with the interaction media is performed at the server side. In the second architecture, the mixing is performed in each client device. The issues of decoding and rendering simultaneous content and interaction media streams on a mobile device are discussed, and the related implementation is presented.

This paper studies the consumer experience for Mobile and Interactive Social TV, which allows users to create an experience of watching together. A focus group study approach was used. The study found that the feeling of social presence of people of interest when watching content with them was considered to add value to the viewing experience. Users found value in availability of all (in this case audio, video, text) interaction modalities. The key requirement is the ability for selective enabling/disabling of individual interaction features as per the user preferences and context. The context was considered to be influenced by both the relation with other participants and the content being consumed. Sports, news, user generated content, short clips were considered to be most suited for social consumption in the mobile context. Battery life, cost and network quality were the major concerns for adopting such a system in a real-world environment.

IP-based Multimedia creation and consumption is becoming available on an increasing spectrum of devices ranging from low-powered portable devices like cell phones and PDA's to high powered static devices like desktops PCs and IPTVs. High-speed wireless and wire-line network access is becoming widespread. Multimedia services like IPTV, Video-on-demand and Video-conferencing are becoming main stream. These developments have paved the way for creating innovative concepts like session mobility for multimedia applications. Session mobility enables seamless transfer of an on-going multimedia session between different devices based on user preferences. We examine the motivations and use cases for session mobility. In this paper we evaluate architectural design constraints and propose architectural solutions for session mobility. We evaluate current session mobility mechanisms and propose requirements for new mechanisms.