The integration of AI/ML technologies into medical imaging is revolutionizing radiology, offering transformative benefits in clinical workflows. AI-powered Software as a Medical Device (SaMD) solutions not only reduce workload and optimize image interpretation but also unlock critical insights previously undetectable by human eyes-catching the unseen and enabling earlier, more accurate diagnoses. Lung cancer, the leading cause of cancer-related mortality worldwide, is often diagnosed at a late stage, when curative treatment is no longer viable. Early detection is paramount. Traditional screening methods rely heavily on nodule size and growth as indicators of malignancy. However, these criteria alone are insufficient for identifying cancer at its earliest, most treatable stage. eyonis® LCS, the flagship clinical development program of Median Technologies, represents a next-generation AI/ML-based SaMD designed specifically for lung cancer screening [1][2]. It combines Computer-Aided Detection (CADe) and Computer-Aided Diagnosis (CADx) [3][4] capabilities to support clinicians in identifying malignant nodules with greater precision. By leveraging specific architectural choices and deep learning models, eyonis® LCS enhances diagnostic accuracy beyond the current standard of care [5], offering a paradigm shift in early lung cancer detection [6]. This presentation will delve into some of the architectural foundations of eyonis® LCS, highlight its clinical impact, and demonstrate how it empowers radiologists to diagnose lung cancer when patients can still be cured. Through this pioneering technology, Median Technologies is redefining the future of cancer screening and patient outcomes.

In recent years, the use of AI/ML technologies in the medical device industry has increased significantly, with a boom in the radiology sector. AI/ML-tech based Software as Medical Devices (SaMD) support healthcare professionals in their clinical routine, saving medical time, decreasing workload while improving image readouts. But more importantly, AI/ML technologies offer the ability to extract valuable and reliable insights from medical images, catching the unseen and paving the way for medical breakthroughs while improving the care pathway for diseases monitored by medical images.

Multi-task learning has gained popularity due to the advantages it provides with respect to resource usage and performance. Nonetheless, the joint optimization of parameters with respect to multiple tasks remains an active research topic. Sub-partitioning the parameters between different tasks has proven to be an efficient way to relax the optimization constraints over the shared weights, may the partitions be disjoint or overlapping. However, one drawback of this approach is that it can weaken the inductive bias generally set up by the joint task optimization. In this work, we present a novel way to partition the parameter space without weakening the inductive bias. Specifically, we propose Maximum Roaming, a method inspired by dropout that randomly varies the parameter partitioning, while forcing them to visit as many tasks as possible at a regulated frequency, so that the network fully adapts to each update. We study the properties of our method through experiments on a variety of visual multi-task data sets. Experimental results suggest that the regularization brought by roaming has more impact on performance than usual partitioning optimization strategies. The overall method is flexible, easily applicable, provides superior regularization and consistently achieves improved performances compared to recent multi-task learning formulations.

Video hyperlinking is a task aiming to enhance the accessibility of large archives, by establishing links between fragments of videos. The links model the aboutness between fragments for efficient traversal of video content. This paper addresses the problem of link construction from the perspective of cross-modal embedding. To this end, a generalized multi-modal auto-encoder is proposed. The encoder learns two embeddings from visual and speech modalities, respectively, whereas each of the embeddings performs self-modal and cross-modal translation of modalities. Furthermore, to preserve the neighbourhood structure of fragments, which is important for video hyperlinking, the auto-encoder is devised to model data distribution of fragments in a dataset. Experiments are conducted on Blip10000 dataset using the anchor fragments provided by TRECVid Video Hyperlinking (LNK) task over the years of 2016 and 2017. This paper shares the empirical insights on a number of issues in cross-modal learning, including the preservation of neighbourhood structure in embedding, model fine-tuning and issue of missing modality, for video hyperlinking.

In recent years, representation learning approaches have disrupted many multimedia computing tasks. Among those approaches, deep convolutional neural networks (CNNs) have notably reached human level expertise on some constrained image classification tasks. Nonetheless, training CNNs from scratch for new task or simply new data turns out to be complex and time-consuming. Recently, transfer learning has emerged as an effective methodology for adapting pre-trained CNNs to new data and classes, by only retraining the last classification layer. This paper focuses on improving this process, in order to better transfer knowledge between CNN architectures for faster trainings in the case of fine tuning for image classification. This is achieved by combining and transfering supplementary weights, based on similarity considerations between source and target classes. The study includes a comparison between semantic and content-based similarities, and highlights increased initial performances and training speed, along with superior long term performances when limited training samples are available.

In this paper, we propose a multimodal framework for video segment interestingness prediction based on the genre and affective impact of movie content. We hypothesize that the emotional characteristic and impact of a video infer its genre, which can in turn be a factor for identifying the perceived interestingness of a particular video segment (shot) within the entire media. Our proposed approach is based on audio-visual deep features for perceptual content analysis. The multimodal content is quantified in a mid-level representation which consists in describing each audio-visual segment as a distribution over various genres (action, drama, horror, romance, sci-fi for now). Some segment might be more characteristic of a media and therefore be more interesting than a segment containing content with a neutral genre. Having determined the genre of individual video segments, we trained a classifier to produce an interestingness factor which is then used to rank segments. We evaluate our approach on the MediaEval2017 Media Interestingness Prediction Task Dataset (PMIT). We demonstrate that our approach outperforms the existing video interestingness approaches on the PMIT dataset in terms of Mean Average Precision.

Associating texts and images is an easy and intuitive task for a human being, but it raises some issues if we want that task to be accomplished by a computer. Among these issues, there is the problem of finding a common representation for images and sentences. Based on recent research about capsule networks, we define a novel model to tackle that issue. This model is trained and compared to other recent models on the Flickr8k database on Image Retrieval and Image Annotation (or Sentence Retrieval) tasks. We propose a new recurrent architecture inspired from capsule networks to replace the traditional LSTM/GRU and show how it leads to improved performances. Moreover, we show that the interest of our model goes beyond its performances and includes its intrinsic characteristics, which can explain why it performs particularly well on the Image Annotation task. In addition, we propose a routing procedure between capsules which is fully learned during the training of our model.

No abstract available.

In the developing of advanced Human-Computer Interaction, multimedia technology plays a fundamental role to increase usability, and accessibility of computer interfaces. The first workshop on Multimedia for Accessible Human Computer Interface (MAHCI) provides a forum to both multimedia and HCI researchers to discuss the accessible human computer interface design, development, and evaluations with the state-of-the-art multimedia technology. It also enables multimedia community to expand its interaction with the HCI industry and broaden the scope of deploying multimedia technology in practical applications. The workshop features 5 papers which cover a number of novel applications and new methodologies in a half day program.

In this paper, we propose a multimodal deep learning architecture for emotion recognition in video regarding our participation to the audio-video based sub-challenge of the Emotion Recognition in the Wild 2017 challenge. Our model combines cues from multiple video modalities, including static facial features, motion patterns related to the evolution of the human expression over time, and audio information. Specifically, it is composed of three sub-networks trained separately: the first and second ones extract static visual features and dynamic patterns through 2D and 3D Convolutional Neural Networks (CNN), while the third one consists in a pretrained audio network which is used to extract useful deep acoustic signals from video. In the audio branch, we also apply Long Short Term Memory (LSTM) networks in order to capture the temporal evolution of the audio features. To identify and exploit possible relationships among different modalities, we propose a fusion network that merges cues from the different modalities in one representation. The proposed architecture outperforms the challenge baselines (38.81 % and 40.47 %): we achieve an accuracy of 50.39 % and 49.92 % respectively on the validation and the testing data.

Second screen applications are becoming key for broadcasters exploiting the convergence of TV and Internet. Authoring such applications however remains costly. In this paper, we present a second screen authoring application that leverages multimedia content analytics and social media monitoring. A back-office is dedicated to easy and fast content ingestion, segmentation, description and enrichment with links to entities and related content. From the back-end, broadcasters can push enriched content to front-end applications providing customers with highlights, entity and content links, overviews of social network, etc. The demonstration operates on political debates ingested during the 2017 French presidential election, enabling insights on the debates.

In this paper, we describe the first-ever machine human collaboration at creating a real movie trailer (officially released by 20th Century Fox). We introduce an intelligent system designed to understand and encode patterns and types of emotions in horror movies that are useful in trailers. We perform multi-modal semantics extraction including audio visual sentiments and scene analysis and employ a statistical approach to model the key defining components that characterize horror movie trailers. The system was applied on a full-length feature film, "Morgan'' released in 2016 where the system identified 10 moments as best candidates for a trailer. We partnered with a professional filmmaker who arranged and edited each of the moments together to construct a comprehensive trailer completing the entire processing as well as the final trailer assembly within 24 hours. We discuss disruptive opportunities for the film industry and the tremendous media impact of the AI trailer. We confirm the effectiveness of our trailer with a very supportive user study. Finally based on our close interaction with the film industry, we also introduce and investigate the novel paradigm of tropes within the context of movies for advancing content creation.

This demo showcases the utility of video hyperlinks with celebrities as the link anchors and their social circles as targets, aiming to help users quickly explore the aboutness of a celebrity by link traversal. Through content analysis, our system embeds hyperlinks into videos such that users can click-and-jump between celebrity faces in different videos to get-to-know their social circles. One peculiar feature is the ability of the system in providing links that maximize users' chance encounter, or serendipitous experience, beyond information need. Our system is enabled by two key components, name-face association and diversity-based ranking, for the aboutness and serendipity features respectively in hyperlinking. The former component assigns names video faces while mining celebrity social networks. The latter performs topic modeling so as to rank videos based on topic diversity.

This paper addresses two key issues which concern the performance of nonparametric scene parsing: (1) the semantic quality of image retrieval; and (2) the accuracy in label transfer. First, because nonparametric methods annotate a query image through transferring labels from retrieved images, the task of image retrieval should find a set of “semantically similar” images to the query. Second, with the retrieval set, a good strategy should be developed to transfer semantic labels in pixel-level accuracy. In this paper, we focus on improving scene parsing accuracy in these two issues. We propose using the state-of-the-art deep convolutional features as image descriptors to improve the semantic quality of retrieved images. In addition, we include dense alignment into the Markov Random Field inference framework to transfer labels at pixel-level accuracy. Our experiments on the SIFT Flow dataset shows the improvement of the proposed approach over other nonparametric methods.

Massive amounts of digital media is being produced and consumed daily on the Internet. Efficient access to relevant information is of key importance in contemporary society. The Hyper Video Browser provides multiple navigation means within the content of a media repository. Our system utilizes the state of the art multimodal content analysis and indexing techniques, at multiple temporal granularity, in order to satisfy the user need by suggesting relevant material. We integrate two intuitive interfaces: for search and browsing through the video archive, and for further hyperlinking to the related content while enjoying some video content. The novelty of this work includes a multi-faceted search and browsing interface for navigating in video collections and the dynamic suggestion of hyperlinks related to a media fragment content, rather than the entire video, being viewed. The approach was evaluated on the MediaEval Search and Hyperlinking task, demonstrating its effectiveness at locating accurately relevant content in a big media archive.

Nowadays, with the continual development of digital capture technologies and social media services, a vast number of media documents are captured and shared online to help attendees record their experience during events. In this paper, we present a method combining semantic inference and multimodal analysis for automatically finding media content to illustrate events using an adaptive probabilistic hypergraph model. In this model, media items are taken as vertices in the weighted hypergraph and the task of enriching media to illustrate events is formulated as a ranking problem. In our method, each hyperedge is constructed using the K-nearest neighbors of a given media document. We also employ a probabilistic representation, which assigns each vertex to a hyperedge in a probabilistic way, to further exploit the correlation among media data. Furthermore, we optimize the hypergraph weights in a regularization framework, which is solved as a second-order cone problem. The approach is initiated by seed media and then used to rank the media documents using a transductive inference process. The results obtained from validating the approach on an event dataset collected from EventMedia demonstrate the effectiveness of the proposed approach.

The twenty papers in this special section aim at providing a forum to present recent advancements in deep learning research that directly concerns the multimedia community. Specifically, deep learning has successfully designed algorithms that can build deep nonlinear representations to mimic how the brain perceives and understands multimodal information, ranging from low-level signals like images and audios, to high-level semantic data like natural language. For multimedia research, it is especially important to develop deep networks to capture the dependencies between different genres of data, building joint deep representation for diverse modalities.

Currently, popular search engines retrieve documents on the basis of text information. However, integrating the visual information with the text-based search for video and image retrieval is still a hot research topic. In this paper, we propose and evaluate a video search framework based on using visual information to enrich the classic text-based search for video retrieval. With the proposed framework, we endeavor to show experimentally, on a set of real world scenarios, that visual cues can effectively contribute to significant quality improvement of video retrieval. Experimental results show that mapping text-based queries to visual concepts improves the performance of the search system. Moreover, when appropriately selecting the relevant visual concepts for a query, a very substantial improvement of the system's performance is achieved.

Currently, popular search engines retrieve documents on the basis of text information. However, integrating the visual information with the text-based search for video and image retrieval is still a hot research topic. In this paper, we propose and evaluate a video search framework based on using visual information to enrich the classic text-based search for video retrieval. The framework extends conventional text-based search by fusing together text and visual scores, obtained from video subtitles (or automatic speech recognition) and visual concept detectors respectively. We attempt to overcome the so called problem of semantic gap by automatically mapping query text to semantic concepts. With the proposed framework, we endeavor to show experimentally, on a set of real world scenarios, that visual cues can effectively contribute to the quality improvement of video retrieval. Experimental results show that mapping text-based queries to visual concepts improves the performance of the search system. Moreover, when appropriately selecting the relevant visual concepts for a query, a very significant improvement of the system's performance is achieved.

This paper introduces a framework for establishing links between related media fragments within a collection of videos. A set of analysis techniques is applied for extracting information from different types of data. Visual-based shot and scene segmentation is performed for defining media fragments at different granularity levels, while visual cues are detected from keyframes of the video via concept detection and optical character recognition (OCR). Keyword extraction is applied on textual data such as the output of OCR, subtitles and metadata. This set of results is used for the automatic identification and linking of related media fragments. The proposed framework exhibited competitive performance in the Video Hyperlinking sub-task of MediaEval 2013, indicating that video scene segmentation can provide more meaningful segments, compared to other decomposition methods, for hyperlinking purposes.

This is the abstract for the ``Video Hyperlinking'' tutorial, presented as part of the 2014 ACM Multimedia Conference. Video hyperlinking is the introduction of links that originate from pieces of video material and point to other relevant content, be it video or any other form of digital content. The tutorial presents the state of the art in video hyperlinking approaches and in relevant enabling technologies, such as video analysis and multimedia indexing and retrieval. Several alternative strategies, based on text, visual and/or audio information are introduced, evaluated and discussed, providing the audience with details on what works and what doesn't on real broadcast material.

With the rapid development of social media sites, a lot of user generated content is being shared in the Web, leading to new challenges for traditional media retrieval techniques. An event describes the happening at a specific time and place in real-world, and it is one of the most important cues for people to recall past memories. The reminder value of an event makes it extremely helpful in organizing human life. Thus, organizing media by events has recently drawn much attention within the multimedia research community. In this paper, we focus on two fundamental problems related to event based social media analysis: the study of feature importance for modeling the relation between events and media, and how to deal with missing and erroneous metadata often present in social media data. These issues are studied within an event-based media classification framework. Different learning approaches are employed to train the event models on different features. We find, through experiments on a large set of events, that the best discriminant features are tags, spatial and temporal feature. We address the missing value problem by extending the feature with an extra attribute to indicate if the values are missing. Promising results are achieved demonstrating the effectiveness of the proposed method.

Exploiting event context to organize social media draws lots of interest from the multimedia community. In this paper, we present our system, called EventEnricher, to infer the semantics behind events and explore social media to illustrate events. We extend the set of illustrating images for a particular event by querying social media with diverse multi-modal features and subsequently pruning the results using content based visual analysis. We integrate the solution into an intelligent interface that enables the user to browse the media collection illustrating events in an easy, effective and informative way.

We present a method combining semantic inferencing and visual analysis for finding automatically media (photos and videos) illustrating events. We report on experiments validating our heuristic for mining media sharing platforms and large event directories in order to mutually enrich the descriptions of the content they host. Our overall goal is to design a web-based environment that allows users to explore and select events, to inspect associated media, and to discover meaningful, surprising or entertaining connections between events, media and people participating in events. We present a large dataset composed of semantic descriptions of events, photos and videos interlinked with the larger Linked Open Data cloud and we show the benefits of using semantic web technologies for integrating multimedia metadata.

The purpose of this workshop is to bring together researchers interested in the construction and analysis of Very Large Scale Multimedia Corpus, as well as the methodologies to Mine and Retrieve information from them. The Workshop will provide a forum to consolidate key issues related to research on very large scale multimedia dataset such as the construction of dataset, creation of ground truth, sharing and extension of such resources in terms of ground truth, features, algorithms and tools etc. The Workshop will discuss and formulate action plan towards these goals.

10.1145/1631272.1631551

Biometric systems have gained the attention of both the research community and the industry becoming an important topic in real application scenarios. Face recognition is, with fingerprint, among the most used techniques since it is natural for humans to recognize people from facial appearance, since the technology is mature, and because, unlike fingerprint, it is completely unintrusive. Existing systems only focus on the appearance of the subjects considering facial expressions as an obstacle to their aim. On the other hand such systems presents several limitations when dealing with variable illumination conditions, head pose, day-to-day variations (e.g. beard, glasses, or make-up), etc. Furthermore, most of the current techniques do not exploit dynamics to detect the liveness of the tested subjects. In this paper we present a study on person recognition from the dynamics of the facial feature points. The aim of this work is to demonstrate that dynamics of facial expressions could be seen as a biometric characteristic. Therefore, only dynamic characteristics are considered and the adopted features are purged of all appearance information. The results clearly show that relevant biometric information can be extracted from facial expressions and other dynamics of the face.

In this paper, we propose a framework to model video sequences using spatiotemporal description of video shots. Spatiotemporal volumes are extracted thanks to an efficient segmentation algorithm. Video shots are described by building an adjacency graph which models the visual properties of the volumes and the spatiotemporal relationships between them. The cost of extracting visual descriptors for the whole shot is reduced by efficiently propagating and merging region descriptors on spatiotemporal volumes. For the comparison of video shots, we propose a similarity measure which tolerates variability in the spatiotemporal representation. Promising experimental results are observed on different visual video shot categories.

Music transcription refers to extraction of a human readable and interpretable description from a recording of a music performance. Automatic music transcription remains, nowadays, a challenging research problem when dealing with polyphonic sounds or when removing certain constraints. Some instruments like guitars and violins add ambiguity to the problem as the same note can be played at different positions. When dealing with guitar music tablature are, often, preferred to the usual music score, as they present information in a more accessible way. Here, we address this issue with a system which uses the visual modality to support traditional audio transcription techniques. The system is composed of four modules which have been implemented and evaluated: a system which tracks the position of the fretboard on a video stream, a system which automatically detects the position of the guitar on the first fret to initialize the first system, a system which detects the position of the hand on the guitar, and finally a system which fuses the visual and audio information to extract a tablature. Results show that this kind of multimodal approach can easily disambiguate 89% of notes in a deterministic way.

This paper presents an automatic semantic concept extraction method which employs low level visual feature fusion. Both static and dynamic feature fusion approaches are studied and evaluated. The main contributions of this paper are: a novel dynamic feature fusion approach inspired from coding is proposed to create compact yet rich signatures; Statistical study of descriptors with and without fusion. To validate and evaluate our approach, we have conducted a set experiments on the classification of soccer video shots. These experiments show, in particular, that the feature fusion step of our system increases the classification rate of 17% comparing to a system without feature fusion.

In this paper, we propose a framework to extend semantic labeling of images to video shot sequences and achieve efficient and semantic-aware spatiotemporal video segmentation. This task faces two major challenges, namely the temporal variations within a video sequence which affect image segmentation and labeling, and the computational cost of region labeling. Guided by these limitations, we design a method where spatiotemporal segmentation and object labeling are coupled to achieve semantic annotation of video shots. An internal graph structure that describes both visual and semantic properties of image and video regions is adopted. The process of spatiotemporal semantic segmentation is subdivided in two stages: Firstly, the video shot is split into small block of frames. Spatiotemporal regions (volumes) are extracted and labeled individually within each block. Then, we iteratively merge consecutive blocks by a matching procedure which considers both semantic and visual properties. Results on real video sequences show the potential of our approach.

This paper proposes an automatic semantic video content indexing and retrieval system based on fusing various low level visual and shape descriptors. Extracted features from region and sub-image blocks segmentation of video shots key-frames are described via IVSM signature (Image Vector Space Model) in order to have a compact and efficient description of the content. Static feature fusion based on averaging and concatenation are introduced to obtain effective signatures. Support Vector Machines (SVM) and neurals network (NNs) are employed to perform classification. The task of the classifiers is to detect the video semantic content. Then, classifiers outputs are fused using neural network based on evidence theory (NN-ET) in order to provide a decision on the content of each shot. The experimental results are conducted in the framework of soccer video feature extraction task1.

In this paper we introduce a new method for fusing classifier outputs. It is inspired from the behavior knowledge space model with the extra ability to work on continuous input values. This property allows to deal with heterogeneous classifiers and in particular it does not require to make any decision at the classifier level. We propose to build a set of units, defining a knowledge space, with respect to classifier output spaces. A new sample is then classified with respect to the unit it belongs to and some statistics computed on each unit. Several methods to create cells and make the final decision are proposed and compared to k-nearest neighbor and decision tree schemas. The evaluation is conducted on the task of video content retrieval which will reveal the efficiency of our approach

In this paper, we propose a new type of multi-dimensional hidden Markov model based on the idea of dependency tree between positions. This simplification leads to an efficient implementation of the re-estimation algorithms, while keeping a mix of horizontal and vertical dependencies between positions. We explain DT-HMM and we present the formulas for the maximum likelihood re-estimation. We illustrate the algorithm by training a 2-dimensional model on a set of coherent images

The GMF4iTV project (Generic Media Framework for Interactive Television) is an IST European project that developed an end-to-end broadcasting platform providing interactivity on heterogeneous multimedia devices such as Set-Top-Boxes, PCs and PDAs according to the Multimedia Home Platform (MHP) part of the DVB standard. The developed platform allows the content providers to create enhanced audiovisual contents with a degree of interactivity at moving object level or shot changes in a video. The end user is then able to interact with moving objects from the video or individual shots allowing the enjoyment of additional contents associated to them (MHP applications, HTML pages, JPEG, MPEG-4 files,...).

The GMF4iTV project is building an end-to-end broadcast system for providing interactivity to TV programs through active video objects. This paper describes the personalization scheme that has been defined and implemented in this project. On the authoring side, an interface tool allows the video producers to annotate both objects and associated additional content, with attributes from different ontologies. On the receiver side, a personalization engine compares these attributes with the user profile to personalize the interaction with the user. We describe the various components of the system and provide some application examples.

Low-level features are now becoming insufficient to build efficient content-based retrieval systems. The interest of users is not any more to retrieve visually similar content, but they expect retrieval systems to find documents with similar semantic content. Bridging the gap between low-level features and semantic content is a challenging task necessary for future retrieval systems. Latent semantic indexing (LSI) was successfully introduced to efficiently index text documents. We propose to adapt this technique to efficiently represent the visual content of video shots for semantic content detection. Although we restrict our approach to visual features, it can be extended with minor changes to audio and motion features to build a multi-modal system. The semantic content is then detected thanks to two classifiers: k-nearest neighbors and neural network classifiers. Finally, in the experimental section we show the performances of each classifier and the performance gain obtained with LSI features compared to traditional features.

Automatically generated summaries of TV content are indispensable for content selection and navigation in CE-devices. We show two different types of summaries: Short video trailers and visual overviews consisting of representative frames. The demo does not only show the feasibility of the proposed algorithms, but also shows how different types of generated summaries can be used in future CE-devices.

This paper presents the evaluation of a number of algorithm alternatives for content based retrieval from a database of technical drawings representing patents. The objective is to help patent evaluators in their quest for a possible patent bearing too much similarity with the one under investigation. To achieve this, we have devised a system where images (drawings) are represented using attributed graphs based on the extracted line-patterns or histograms of attributes computed from the graphs. Retrieval is either performed using histogram comparison (see Huet, B. and Hancock, E.R., PAMI, vol.21, no.12, p.1363-70, 1999) or thanks to a graph similarity measure (see Huet and Hancock, CVPR, p.138-43, 1998). Promising results are presented along with possible work extension.

The growing availability of multimedia data such as video on personal computers and home equipment creates a strong requirement for efficient tools to manipulate this type of data. Automatic summarization is one of these tools, which automatically creates a short version or subset of key-frames which contains as much information as possible as in the original video. Several approaches have been proposed to define and identify what is the most important content in a video. In this paper, we propose a new approach for the automatic creation of summaries for multi-episode videos, such as TV series. In this case, it is necessary to identify similarities and differences among videos (what’s common, what’s unique, how they differ) in order to find which elements best characterize a particular video with respect to the others. We describe our proposed method, provide the results of its application on a sample set of videos, and suggest a new criterion to evaluate the quality of the summaries that have been created.

This paper presents a comparison of some methodologies for the automatic construction of video summaries. The work is based on the simulated user principle to evaluate the quality of a video summary in a way which is automatic, yet related to user's perception. The method is studied for the case of multi-episode video, where we do not only describe what is important in a video, but rather what distinguishes this video from the others. Experimental results are presented to support the proposed ideas.

While one-dimensional hidden Markov models have been very successfully applied to numerous problems, their extension to two dimensions has been shown to be exponentially complex, and this has very much restricted their usage for problems such as image analysis. In this paper we propose a novel algorithm which is able to approximate the search for the best state path (Viterbi decoding) in a 2D HMM. This algorithm makes certain assumptions which lead to tractable computations, at a price of loss in full optimality. We detail our algorithm, its implementation, and present some experiments on handwritten character recognition. Because the Viterbi algorithm serves as a basis for many applications, and 1D HMMs have shown great flexibility in their usage, our approach has the potential to make 2D HMMs as useful for 2D data as 1D HMMs are for 1D data such as speech.

The paper studies the structural sensitivity of line pattern recognition using shape graphs. We compare the recognition performance for four different algorithms. Each algorithm uses a set of pairwise geometric attributes and a neighbourhood graph to represent the structure of the line patterns. The first algorithm uses a pairwise geometric histogram, the second uses a relational histogram on the edges of the shape graph, the third compares the set of attributes on the edges of the shape graph and the final algorithm compares the arrangement of line correspondences using graph matching. The different algorithms are compared under line deletion, line addition, line fragmentation and line end point measurement errors. It is the graph matching algorithm which proves to be the most effective.

This paper presents a new compact shape representation for retrieving line-patterns from large databases. The basic idea is to exploit both geometric attributes and structural information to construct a shape histogram. We realize this goal by computing the N-nearest neighbor graph for the lines-segments for each pattern. The edges of the neighborhood graphs are used to gate contributions to a two-dimensional pairwise geometric histogram. Shapes are indexed by searching for the line-pattern that maximizes the cross correlation of the normalized histogram bin-contents. We evaluate the new method on a database containing over 2,500 line-patterns each composed of hundreds of lines.

This paper investigates the use of shape-from-shading for object recognition. The local surface orientation information recovered using shape-fromshading is shown to provide useful input to an appearance-based object recognition scheme. We consider two representations which may be recovered from shading information- the needle-map, and the local curvature shapeindex- and examine their relative performance for object recognition. Specifically, we use a histogram-comparison technique, and focus upon the relative stability of the representations to small changes of viewpoint. We demonstrate that the needle-map representation allows the view-sphere to be spanned using a significantly smaller number of characteristic views than using either the raw images or the shape index. 1

This paper presents a new similarity measure for object recognition from large libraries of line-patterns. The measure draws its inspiration from both the Hausdorff distance and a recently reported Bayesian consistency measure that has been successfully used for graph-based correspondence matching. The measure uses robust error-kernels to gauge the similarity of pair-wise attribute relations defined on the edges of nearest neighbour graphs. We use the similarity measure in a recognition experiment which involves a library of over 1000 line-patterns. A sensitivity study reveals that the method is capable of delivering a recognition accuracy of 98%. A comparative study reveals that the method is most effective when a Gaussian kernel or Huber's robust kernel is used to weight the attribute relations. Moreover, the method consistently outperforms Rucklidge's median Hausdorff distance (1995).

This paper is concerned with the retrieval of images from large databases based on their shape similarity to a query image. Our approach is based on two dimensional histograms that encode both the local and global geometric properties of the shapes. The pairwise attributes are the directed segment relative angle and directed relative position. The novelty of the proposed approach is to simultaneously use the relational and structural constraints, derived from an adjacency graph, to gate histogram contributions. We investigate the retrieval capabilities of the method for various queries. We also investigate the robustness of the method to segmentation errors. We conclude that a relational histogram of pairwise segment attributes presents a very efficient way of indexing into large databases. The optimal configuration is obtained when the local features are constructed from six neighbouring segments pairs. Moreover, a sensitivity analysis reveals that segmentation errors do not affect the retrieval performances.

This paper investigates the use of shape-from-shading for object recognition. The local surface orientation information recovered using shape-from-shading is shown to provide useful input to an appearance-based object recognition scheme. We consider two representations which may be recovered from shading information-the needle-map, and the local curvature shape-index-and their relative performance for object recognition. Specifically, we use a histogram-comparison technique, and focus upon the relative stability of the representations to small changes of viewpoint. We demonstrate that the needle-map representation allows the view-sphere to be spanned using a significantly smaller number of characteristic views than using either the raw images or the shape index.

The paper aims to develop simple statistical methods for indexing line patterns. The application vehicle used in this study involves indexing into an aerial image database using a cartographic model. The images contained in the database are of urban and semi urban areas. The cartographic model represents a road network known to appear in a subset of the images contained within the database. There are known to be severe imaging distortions present and the data cannot be recovered by applying a simple Euclidean transform to the model. We effect the cartographic indexing into the database using pairwise histograms of the angle differences and the cross ratios of the lengths of line segments extracted from the raw aerial images. We investigate several alternative ways of performing histogram comparison. Our conclusion is that the Matusita and Bhattachargya distances provide significant performance advantages over the L/sub 2/ norm employed by M. Swain and D. Ballard (1990). Moreover, a sensitivity analysis reveals that the angle difference histogram provides the most discriminating index of line structure; it is robust both to image distortion on to the variable quality of input line segmentation.