This survey introduces the current state of the art in image and video retargeting and describes important ideas and technologies that have influenced the recent work. Retargeting is the process of adapting an image or video from one screen resolution to another to fit different displays, for example, when watching a wide screen movie on a normal television screen or a mobile device. As there has been considerable work done in this field already, this survey provides an overview of the techniques. It is meant to be a starting point for new research in the field. We include explanations of basic terms and operators, as well as the basic workflow of the different methods.

Many teachers use Audience Response Systems (ARS) in lectures to re-activate their listeners and to get an insight in students’ knowledge of the current lecture contents. Plenty of such applications have been developed in recent years, they provide a high variety of different teaching scenarios with the use of the students’ smartphones, including quizzes, lecture feedback and dynamic message boards. We developed a novel application based on an abstract model to enable this variety of customizable teaching scenarios within one application. After presenting the application to the first charge of lecturers, the responses were quite good, and several new teaching scenarios were created and used. This paper presents first experiences when using a variety of customizable teaching scenarios, the special opportunities and challenges as well as the opinions of lecturers and students, which we collected with a survey at the end of the semester.

With the increasing number of shops and service providers offering loyalty cards and attractive benefits for card holders, the amount of different cards a typical customer owns has become a problem. In most cases, a card needs to be scanned in the particular shop to gain discounts meaning the customers have to always carry a large number of cards. In this paper, we present an approach to automatically detect the card provider, card type and customer number, which are then stored within a mobile application, allowing customers to carry all their cards with them on their smartphone. We propose ACC* as a new evaluation metric and evaluate the performance of this approach on a large real-world dataset of loyalty cards.

This work discusses a distributed interactive video system that supports video annotation using simultaneous hyperlinking by multiple users. The users mark and annotate objects within the video with links to other media such as text, images, websites, or other videos. Annotations are visualized on the client user interface as an overlay close to the objects. Our system is intuitive to use; for example, it contains automatic object-tracking functionality that correctly positions the annotations, even when the form or location of an object changes. Thus, our first contribution discusses the adaptive object-tracking algorithm used for this repositioning. It shows improved precision and reliability in comparison to nonadaptive algorithms. A second key issue is to keep the system responsive when the number of concurrent annotators increases. Thus, we rely on the concept of eventual consistency between different network entities. While this weak form of consistency allows temporary inconsistencies, it ensures that a consistent state can be reached. Thus, the second contribution is the design and evaluation of our distributed interactive video system, which relies on the weak consistency paradigm.

Audience Response Systems (ARS) are quite common nowadays. With a very high smart phone availability among students, the usage of ARS within classroom settings has become quite easy. Together with the trend for developing web applications, the number of ARS implementations grew rapidly in recent years. Many of these applications are quite similar to each other, and fit into many classroom learning scenarios like test questions, self-assessment and audience feedback. But they are mostly limited to their original purpose. Using another question types or little differences in the existing quiz logic cause considerable effort to develop as they have to be implemented separately. We searched for similarities between the different ARS implementations, matched them to a universal process and present a generalized model for all of them. We implemented a prototype that serves many known scenarios ranging from simple knowledge feedback questions up to complex marketplace simulations. A first evaluation in different course types with up to fifty students showed, that the model satisfies our expectations and offers a lot of new opportunities for classroom learning scenarios.

The re-capturing of video content poses significant challenges to algorithms in the fields of video forensics, watermarking, and near-duplicate detection. Using a camera to record a video from a display introduces a variety of artifacts, such as geometric distortions, luminance transformations, and temporal aliasing. A deep understanding of the causes and effects of such phenomena is required for their simulation, and for making the affected algorithms more robust. In this paper, we provide a detailed model of the temporal effects in re-captured video. Such effects typically result in the re-captured frames being a blend of the original video's source frames, where the specific blend ratios are difficult to predict. Our proposed parametric model captures the temporal artifacts introduced by interactions between the video renderer, display device, and camera. The validity of our model is demonstrated through experiments with real re-captured videos containing specially marked frames.

To compose high-quality movies directors need life-long learning and talent. User-generated video defines a new era of video production in which non-professionals record videos and share them on platforms such as YouTube. As hiring professional directors results in high costs, our work focuses on replacing those directors by crowdsourcing. The proposed system allows users to record and stream live videos to servers on which workers create a video mashup. A smartphone application for recording live video has been designed that supports the composition in the crowd by a multi-modal analysis of the recording quality. The contributions of this work are: The proposed system demonstrates that composing a large number of video views can be achieved in near real-time. Second, the system achieves comparable video quality for user-generated video in comparison to manual composition. Third, it offers insights on how to design real-time capable crowdsourcing systems. Fourth, by leveraging multi-modal features that can already be evaluated during recording the number of streams considered for presentation can be reduced.

To support the development of any system that includes the generation and evaluation of camcorder copies, as well as to provide a common benchmark for robustness against camcorder copies, we present a tool to simulate digital video re-acquisition using a digital video camera. By resampling each video frame, we simulate the typical artifacts occurring in a camcorder copy: geometric modifications (aspect ratio changes, cropping, perspective and lens distortion), temporal sampling artifacts (due to different frame rates, shutter speeds, rolling shutters, or playback), spatial and color subsampling (rescaling, filtering, Bayer color filter array), and processing steps (automatic gain control, automatic white balance). We also support the simulation of camera movement (e.g., a hand-held camera) and background insertion. Furthermore, we allow for an easy setup and calibration of all the simulated artifacts, using sample/reference pairs of images and videos. Specifically temporal subsampling effects are analyzed in detail to create realistic frame blending artifacts in the simulated copies. We carefully evaluated our entire camcorder simulation system and found that the models we developed describe and match the real artifacts quite well.

In this paper, a novel warping-based retargeting approach for stereoscopic video is proposed. It considers the three conflicting goals: preserving salient image content, avoiding flickering and maintaining consistency between the two views. The first step of the approach focuses on content-aware image resizing and considers image saliency, motion saliency, and depth saliency. In the second step, temporal coherence is preserved by tracking and optimizing deformed pathlines. The proposed algorithm maps the mesh from the left to the right view to guarantee consistency between the deformation of objects between the views. Users rated the quality of the adapted videos as good. In particular, distortions in the perceived depth are not noticeable, and the temporal stability is significantly higher compared to seam carving approaches.

In this paper, we present a fast parallel algorithm for the retargeting of videos. It combines seam carving and cropping and is aimed for real-time adaptation of video streams. The basic idea is to first find an optimal cropping path over the whole sequence with the target size. Then, the borders are slightly extended to be reduced again by seam carving on a frame-by-frame basis. This allows the algorithm to get more important content into the cropping window as it is also able to remove pixels from within the window. In contrast to the previous SeamCrop algorithm, the presented technique is optimized for parallel processes and a CUDA GPU implementation. In comparison, the computation time of our GPU algorithm is 10.5 times faster (on a 960 x 540 video with a retarget factor of 25%) than the already efficient CPU implementation.

In 1998, Petitcolas et al. proposed StirMark [14] as a benchmark for image watermarking schemes. The main idea was to introduce a re-sampling process that mimics the analog process of printing and scanning a watermarked image. For digital video, the corresponding concept is a camcorder copy, where a video displayed on a screen is (digitally) recorded using a video camera. As most commercial video streaming systems (VOD, IPTV) and offline distribution (Blu-ray, HDDs for cinemas) are strongly protected by means of DRM, filming a display is actually a relevant use case and a requirement for robust video watermarking systems to survive.

We present a novel system to assist users in choosing suitable exposure values for photography on smartphones. The user specifies the desired shape of the histogram of the captured image by adjusting the parameters of a function. The system then uses the smartphone's camera and varying exposure values to calculate a high dynamic range (HDR) histogram of the scene. It contains the entire brightness range from the darkest to the brightest area of the scene. For any exposure value, the HDR histogram can be used to synthesize a conventional histogram of a low dynamic range (LDR) image as if captured by this exposure value. This process is much faster than capturing the image itself. By comparing a number of synthesized LDR histograms to the user's desired histogram shape, the system determines an exposure value that best meets the user's preference.

We present a novel system for automatically detecting salient image regions in stereoscopic videos. Our proposed algorithm considers information based on three dimensions: salient colors in individual frames, salient information derived from camera and object motion, and depth saliency. These three components are dynamically combined into one final saliency map based on the reliability of the individual saliency detectors. Such a combination allows using more efficient algorithms even if the quality of one detector degrades. For example, we use a computationally efficient stereo correspondence algorithm that might cause noisy disparity maps for certain scenarios. In this case, however, a more reliable saliency detection algorithm such as the image saliency is preferred. To evaluate the quality of the saliency detection, we created modified versions of stereoscopic videos with the non-salient regions blurred. Having users rate the quality of these videos, the results show that most users do not detect the blurred regions and that the automatic saliency detection is very reliable.

Although the importance of video sharing and of social media is increasing from day to day, a full integration of videos into social media is not achieved yet. We have developed a system that maps the concept of hypervideo - allowing to annotate objects in a video - to social media. We define this combination as social video that simultaneously allows a large number of users to contribute to the content of a video. Users can annotate video objects by adding images, text, other videos, Web links, or even communication topics. An integrated chat system allows users to communicate with friends and to link these topics to distinct objects in the video. We analyze the technical functionality and the user acceptance of our social video system in detail. Due to the integration into the social network Facebook more than 12,000 users have already accessed our system.

We present a system that is capable of capturing, processing and displaying High Dynamic Range (HDR) videos at 23 frames per second. For each frame in the video, a sequence of regular images is captured in quick succession at optimally chosen shutter speeds. Intermediate camera motion is compensated and the images are combined into an HDR frame which is tone mapped for display. Our system makes use of novel algorithms that are fast enough for processing the frames in real-time. The most time-consuming operations are parallelized and executed on a graphics card for a speed-up of 15 to 1 over the sequential version.

We present a novel system that allows small handheld devices to detect other devices in the local proximity and to exchange messages with them. The client devices use WLAN to access the system via a web interface and do not require additional software or modification of the device's operating system. The physical closeness of client devices is estimated based on their signal strengths measured by the system's access points. A major advantage compared to indoor positioning systems is the fact that our system does not require a training phase. We conduct a comprehensive evaluation to demonstrate the practicability of our approach.

We propose a novel approach for tracking video objects in interactive multimedia systems. Instead of designing a single tracking algorithm that works well with some templates (video objects to be tracked) but fails with others, our adaptive algorithm uses three distinct tracking techniques with different strengths and weaknesses. The tracking technique that is selected for a given template depends on its visual content and the content of the video. Our approach is robust and allows tracking requests of users to be computed despite an imprecise selection of a template. In addition, our approach is designed to comply with high precision, speed, and scalability to support many users simultaneously. As an exemplary testing environment for this tracking system, an interactive hypervideo system was chosen. The system was integrated into the social network Facebook and used by more than 12,000 users.

The quality achieved by simply scaling a sports video to the limited display resolution of a mobile device is often insufficient. As a consequence, small details like the ball or lines on the playing field become unrecognizable. In this paper, we present a novel approach to analyzing court-based ball sports videos. We have developed new techniques to distinguish actual playing frames, to detect players, and to track the ball. This information is used for advanced video retargeting, which emphasizes essential content in the adapted videos. We evaluate the precision and recall achieved in the analysis and measure the computational time taken. In addition, we compare our new approach to other video retargeting techniques based on scaling, cropping, and seam carving.

We introduce a novel approach for image registration for high dynamic range (HDR) videos. We estimate a translation vector between two low dynamic range (LDR) frames captured at different exposure settings. By using row and column histograms, counting the number of dark and bright pixels in a row or column, and maximizing the correlation between the histograms of two consecutive frames, we reduce the two-dimensional problem to two one-dimensional searches. This saves computation time, which is critical in recording HDR videos in real-time. The robustness of our estimation is increased through application of a Kalman filter. A novel certainty criterium controls whether the estimated translation is used directly or discarded and extrapolated from previous frames. Our experiments show that our proposed approach performs registration more robustly on videos and is 1.4 to 3 times faster than comparable algorithms.

In the efforts to fight piracy of high-valued media content, forensic digital watermarking as a passive content security scheme is a potential alternative to current, restrictive approaches like DRM. In this paper, we present a novel watermarking scheme for videos based on affine geometric transformations. Frames can be modified in an imperceptible manner by applying a small, global rotation, translation, or zooming, which can be detected later on by comparison with the originals. To compensate geometric distortions that have been introduced while a video travels down legal as well as illegal distribution chains, a spatio-temporal synchronization is performed using our video registration toolkit application. To evaluate our approach, we compare it with several other schemes regarding the robustness against common attacks, including camcorder capture.

Virtually every video watermarking technology can benefit from comparison with the original content. For non-blind schemes it is fundamental; for others it is an improvement to increase the watermark’s signal-to-noise ratio by subtract-ing the content that is often noise to the detector. A direct frame-by-frame comparison of the videos is not possible due to the fact that illegal copies of videos usually differ signifi-cantly from their originals caused by different spatial resolu-tion or frame rates, geometric distortions from capturing, or targeted attacks. In this paper, we present a software tool that enables the semi-automatic temporal and spatial syn-chronization of frames and pixels of two similar videos. This process is called registration. We put our focus on utilizing human capabilities with the smallest possible effort, to allow a high overall performance and precision of the registration. An efficient graphical user interface supports the users and visualizes the results of all steps. In addition, we specifically distinguish digitally reproduced copies from analog (cam-corded) copies in which two or more frames are blended into a new frame.

The presentation of multimedia data and especially of high resolution videos on small mobile devices is still a great challenge today. Both cropping of borders and scaling of frames may result in the removal of essential content of videos or lost details due to the reduced size of the visual content. Another major problem emerges if the aspect ratio of the original video and the display of the mobile device differ. User evaluations indicate that changing the aspect ratio may reduce the visual quality of videos significantly. In this paper, we present the new FSCAV algorithm (Fast Seam Carving for Size Adaptation of Videos) to adapt the size of videos to the limited display resolution and different aspect ratios of handheld mobile devices. The general idea of the seam carving algorithm for still images is to remove seams in images so that the essential content is preserved. We extended this technique which works very well for images to create videos without jitter or visible artifacts. A major feature of our FSCAV algorithm is the low computational complexity which enables an efficient adaptation of videos to small screens. Nevertheless, severe distortions are clearly visible in some shots of the adapted videos. We present a new heuristic to identify shots with such a low visual quality. If the quality drops below a threshold, a different adaptation technique is used for this shot (e.g., scaling or cropping). User evaluations confirm a very high visual quality of our approach.

We present our system for automatic lecture recording. In contrast to traditional lecture recording systems our approach aims to imitate a real camera team, consisting of multiple cameramen and a director, in order to make the recording more vivid. At first, we introduce the environment. Then, we present our approach, beginning with the director module, followed by the cameraman module and the sensor tools module, referring to the jobs of the respective human originals. The director is based on an Extended Finite State Machine, with transition conditions depending on input from sensor tools. The cameraman automatically controls iris, focus, etc., and also takes basic cinematographic rules into account. The video concludes with an example result of our virtual camera team of a real lecture. A video demo of our approach is available at [4].

In this paper we present an optimized approach to capture high dynamic range (HDR) images. It is based on existing methods of creating HDR images by fusing a set of differently exposed low dynamic range (LDR) images. We optimize the capturing process of LDR images towards improved capture speed by using partial re-exposures. That is, we make use of the idea that it is not always necessary to capture full size images when only small portions of the scene require HDR. By analyzing captured images for badly exposed regions and re-exposing selectively, we save overall capture time and increase the frame rate when image sequences are recorded.

A challenge for the development of mobile applications is the heterogeneity of the different device classes. For instance, an application which runs well on a personal digital assistant (PDA) might be unsuitable for mobile phones. To facilitate the development of mobile applications we have develope

While indoor positioning systems based on 802.11 and fingerprinting work pretty well, it is unknown how to distribute large amounts of fingerprint data to mobile devices. Even worse, many mobile devices are restricted in terms of memory capabilites. We identified three classes of mobile devices representing different levels of storage capabilities. For each of these classes, we present a distribution approach for fingerprint data: the Union of Access Points (UAP), the Strongest Access Point (SAP), and the Intersection of Access Points (IAP) algorithm. These approaches utilize the 802.11 network to download only a subset of the fingerprint data to a mobile device. The subset covers the area close to the actual position of the mobile device in such a way that position estimates can be computed. The size of the subset is different for each of the three algorithms.

Not only the communication capabilities of 802.11, but also the capability to determine the position of mobile devices make 802.11 highly appealing for many application areas. Typically, a mobile device that wants to identify its position regularly performs active or passive scans to obtain the signal strength measurements of neighboring access points. However, so far, no investigations are known to have been launched into how regular scanning affects concurrent data transmissions from an end-user point of view. In this paper, we explore how common data communication is affected while actively or passively scanning at the same time. Furthermore, we present a novel scan scheme called monitor sniffing. Monitor sniffing exploits the fact that 802.11 operates on overlapping channels by overhearing the wireless interface. We have implemented our monitor sniffing algorithm using commodity 802.11g hardware, and we demonstrate that it does not disturb concurrent data communication

A large number of videos cannot be visualized on mobile devices (e.g.,PDAs or mobile phones)due to an inappropriate color depth of the displays. Important details are lost if the color depth is reduced. A major challenge is the preservation of the semantic content in spite of this fact. In this paper, we present a novel adaptation algorithm to enable the playback of videos on color-limited mobile devices. Dithering algorithms diffuse the error to neighbor pixels and do not work very well for videos. We propose a non-linear transformation of luminance values and use textures in combination with edges to reduce the color depth in videos.

A large number of previously recorded videos cannot be directly visualized on mobile devices like PDAs or mobile phones due to an inappropriate screen resolution of their displays. Transcoding can be used to change the resolution, however, the usual transcoding algorithms have problems preserving the semantic content. For instance, superimposed text is unreadable if the character size drops below a certain value. In this paper, we present a novel adaptation algorithm to scale and crop videos while preserving their semantic content. Semantic features in a shot are combined to select a suitable region to be resented in the adapted video.

The curvature scale space (CSS) technique, which is also part of the MPEG-7 standard, is a robust method to describe complex shapes. The central idea is to analyze the curvature of a shape and derive features from inflection points. A major drawback of the CSS method is its poor representation of convex segments: Convex objects cannot be represented at all due to missing inflection points. We have extended the CSS approach to generate feature points for concave and convex segments of a shape. This generic approach is applicable to arbitrary objects. In the experimental results, we evaluate as a comprehensive example the automatic recognition of characters in images and videos.

A video summary is a sequence of video clips extracted from a longer video. Much shorter than the original, the summary preserves its essential messages. In the ECHO (European Chronicles On-line) project, a system was developed to store and manage large collections of historical films for the preservation of cultural heritage. At the University of Mannheim, we have developed the video summarization component of the ECHO system. We discuss the particular challenges the historical film material poses, and how we have designed new video processing algorithms and modified existing ones to cope with noisy black-and-white films.