Automatic metrics are used as proxies to evaluate abstractive summarization systems when human annotations are too expensive.To be useful, these metrics should be fine-grained, show a high correlation with human annotations, and ideally be independent of reference quality; however, most standard evaluation metrics for summarization are reference-based, and existing reference-free metrics correlate poorly with relevance, especially on summaries of longer documents.In this paper, we introduce a reference-free metric that correlates well with human evaluated relevance, while being very cheap to compute.We show that this metric can also be used alongside reference-based metrics to improve their robustness in low quality reference settings.

Large language models and multimodal language-vision models give impressive results on current available summarization benchmarks, but are not designed to handle long multimodal documents. Most summarization datasets are composed of either mono-modal documents or short multimodal documents. In order to develop models designed for understanding and summarizing real-world videoconference records that are typically around 1 hour long, we propose a dataset of 9,103 videoconference records extracted from the German National Library of Science and Technology (TIB) archive, along with their abstract. Additionally, we process the content using automatic tools in order to provide the transcripts and key frames. Finally, we present experiments for abstractive summarization, to serve as baseline for future research work in multimodal approaches.

The number of video surveillance cameras has increased by a large amount in recent years. There is therefore a need to process the captured videos such that human operators can quickly review the activities recorded by a camera over a long period of time. We propose in this paper an approach for producing video summaries, an abbreviated video preserving the important elements of interest. We introduce a dataset as well as three evaluation metrics for quantifying the performance of a video summary with respect to compression length, the amount of activity retained and the amount of activity that is packed into each frame of the summary.

Saliency models provide heatmaps highlighting the probability of each pixel to attract human gaze. To define image's important regions, features maps are extracted. The rarity, surprise or contrast are computed leading to conspicuity maps, showing important regions of each feature map. The final saliency map is obtained by merging these maps. The fusion process is usually a linear combination of the maps where the coefficients show their importance. We propose a novel generic fusion mechanism based on 1) using a rarity-based attention module and 2) using neural networks to achieve the fusion. The first layer of the NN merges the weighted feature maps into a saliency map. The second layer takes into account the spatial information. The approach is compared to 8 models using 4 different comparison metrics on open state-of-the-art databases.

Modeling human attention has been arousing a lot of interest due to its numerous applications. The process that allows us to focus on some more important stimuli is defined as the “attention”. Seam carving is an approach to resize images or video sequences while preserving the semantic content. To define what is important, gradient was first used but due to limitations of this approach, saliency models, which are able to predict whether regions in the images attract human attention, are now used. Most of them are optimized to conform a ground truth like eye tracking but there is no way to know the efficiency of the saliency models applied in a specific application like in seam carving. In this paper, we propose a criterion, based on the quantity of geometric deformation and the image's reduction, which evaluate the quality of a resizing by seam carving. This criterion is applied on evaluation of image (SCES) or video (SCED) resizing. We validate our criterion with subjective evaluation and used it to rank state of the art saliency models for seam carving. Evaluation of the image by SCES gives a Spearman correlation of −0.92196 and a Pearson correlation of −0.8812. For the video, the final SCED gives a Spearman correlation of −0.81351 and a Pearson correlation of −0.80581.

In this paper, a new spatio-temporal saliency model is presented. Based on the idea that both spatial and temporal features are needed to determine the saliency of a video, this model builds upon the fact that locally contrasted and globally rare features are salient. The features used in the model are both spatial (color and orientations) and temporal (motion amplitude and direction) at several scales. To be more robust to moving camera a module computes the global motion and to be more consistent in time, the saliency maps are combined together after a temporal filtering. The model is evaluated on a dataset of 24 videos split into 5 categories (Abnormal, Surveillance, Crowds, Moving camera, and Noisy). This model achieves better performance when compared to several state-of-the-art saliency models.

We propose a full reference visual quality metric to evaluate a semantic coding system which may not preserve exactly the position and/or the shape of objects. The metric is based on Scale-Invariant Feature Transform (SIFT) points. More specifically, Structural SIMilarity (SSIM) on windows around the SIFT points measures the compression artifacts (SSIM_SIFT). Conversely, the standard deviation of the matching distance between the SIFT points measures the geometric distortion (GEOMETRIC_SIFT). We validate our metric with subjective evaluation and reach a Spearman correlation of 0.86 for SSIM_SIFT and 0.74 for GEOMETRIC_SIFT.

Traditional video codecs like H.264/AVC encode video sequences to minimize the Mean Squared Error (MSE)at a given bitrate. Seam carving is a content-aware resizing method. In this paper, we propose a semantic video compression scheme based on seam carving. Its principle is to suppress non salient parts of the video by seam carving. The reduced sequence is then encoded with H.264/AVC and the seams are represented and encoded with our proposed approach. The main idea is to encode the seams by regrouping them. Compared to our earlier work, the main contributions of this paper are: a new energy map with better temporal robustness, a new way to define groups of seams using k-median clustering, and an improved background synthesis. Experiments show that, compared to a traditional H.264/AVC encoding, we reach a bitrate saving between 10% and 24%%with the same quality of the salient objects.