Abstract One of the powers of visual media lies in its ability to create a lasting impression on the viewer’s memory. In this digital age, where media is abundant and attention spans are fleeting, the task of predicting which content will stick in the viewer’s mind has become a critical challenge in computer vision. Computational memorability seeks to address this by developing models that estimate how memorable a piece of media is likely to be. In this review we focus on the MediaEval Predicting Video Memorability benchmark, a recurring evaluation task that has run annually since 2018. This benchmark provides a unique and consistent framework for researchers to compare and refine their memorability prediction techniques using standardised datasets and metrics. Its reproducible framework has proven invaluable for tracking progress and fostering innovation in this rapidly evolving domain. We analyse the evolution of the benchmark across its 2018–2023 editions, discussing the challenges that still remain, such as the need for more interpretability in models and the difficulty of predicting subjective and context-dependent memorability. By analysing and synthesising the collective insights gained from this task, we endeavour to inspire new avenues of inquiry and drive progress towards a more comprehensive understanding of this topic.

LLMs and MM-LLMs may be about to bring a wrecking ball to our past decades of work in multimedia analysis and retrieval by nullifying it and replacing it by an approach of fine-tuning a foundational model. This panel explores the issue of whether the new approach will wreck or build upon our older work, leading to better all round outcomes?

Models of computational memorability have historically been predicated on "yes/no" recognition memory games, resultantly overlooking and obscuring the variability in how we remember-from unprompted intentional detail oriented retrieval to prompted feeling based familiarity. In this paper, we detail an innovative short-term video memorability experiment which leverages drawings as a measure of recollection to explore the relationship between recognition and recall memorability of a previously-viewed video, finding evidence to suggest a measurable interaction. Our findings highlight the need to refine how we currently quantify of remembrance to more faithfully reflect its true phenomenology, and accordingly adjust our current computational models of memorability so that their downstream application in multimedia retrieval may be of higher utility.

Handwriting is an art form that most people learn at an early age. Each person’s writing style is unique with small changes as we grow older and as our mood changes. Here we analyse handwritten text in a culturally significant personal diary. We compare changes in handwriting and relate this to the sentiment of the written material and to the topic of diary entries. We identify handwritten text from digitised images and generate a canonical form for words using shape matching to compare how the same handwritten word appears over a period of time. For determining the sentiment of diary entries, we use the Hedonometer, a dictionary-based approach to scoring sentiment. We apply these techniques to the historical diary entries of Rosamond Jacob (1888-1960), an Irish writer and political activist whose daily diary entries report on the major events in Ireland during the first half of the last century.

In a world of ephemeral moments, our brain diligently sieves through a cascade of experiences, like a skilled gold prospector searching for precious nuggets amidst the river’s relentless flow. This study delves into the elusive "moment of memorability"—a fleeting, yet vital instant where experiences are prioritised for consolidation in our memory. By transforming subjects’ encoding phase electroencephalography (EEG) signals into the visual domain using scaleograms and leveraging deep learning techniques, we investigate the neural signatures that underpin this moment, with the aim of predicting subject-specific recognition of video. Our findings not only support the involvement of theta band (4-8Hz) oscillations over the right temporal lobe in the encoding of declarative memory, but also support the existence of a distinct moment of memorability, akin to the gold nuggets that define our personal river of experiences.

Event cameras, also known as neuromorphic cameras, are an emerging technology that offer advantages over traditional shutter and frame-based cameras, including high temporal resolution, low power consumption, and selective data acquisition. In this study we harnesses the capabilities of event-based cameras to capture subtle changes in the surface of the skin caused by the pulsatile flow of blood in the wrist region. We show how an event camera can be used for continuous non-invasive monitoring of heart rate (HR). Event camera video data from 25 participants with varying age groups and skin colours, was collected and analysed. Ground-truth HR measurements were used to evaluate of the accuracy of automatic detection of HR from event camera data. Our results demonstrate the feasibility of using event cameras for HR detection.

The task of Generic Event Boundary Detection (GEBD) aims to detect moments in videos that are naturally perceived by humans as generic and taxonomy-free event boundaries. Modeling the dynamically evolving temporal and spatial changes in a video makes GEBD a difficult problem to solve. Existing approaches involve very complex and sophisticated pipelines in terms of architectural design choices, hence creating a need for more straightforward and simplified approaches. In this work, we address this issue by revisiting a simple and effective self-supervised method and augment it with a differentiable motion feature learning module to tackle the spatial and temporal diversities in the GEBD task. We perform extensive experiments on the challenging Kinetics-GEBD and TAPOS datasets to demonstrate the efficacy of the proposed approach compared to the other self-supervised state-of-the-art methods. We also show that this simple self-supervised approach learns motion features without any explicit motion-specific pretext task. Our results can be reproduced on $github$.

Recent developments in in-situ monitoring and process control in Additive Manufacturing (AM), also known as 3D-printing, allows the collection of large amounts of emission data during the build process of the parts being manufactured. This data can be used as input into 3D and 2D representations of the 3D-printed parts. However the analysis and use, as well as the characterization of this data still remains a manual process. The aim of this paper is to propose an adaptive human-in-the-loop approach using Machine Learning techniques that automatically inspect and annotate the emissions data generated during the AM process. More specifically, this paper will look at two scenarios: firstly, using convolutional neural networks (CNNs) to automatically inspect and classify emission data collected by in-situ monitoring and secondly, applying Active Learning techniques to the developed classification model to construct a human-in-the-loop mechanism in order to accelerate the labeling process of the emission data. The CNN-based approach relies on transfer learning and fine-tuning, which makes the approach applicable to other industrial image patterns. The adaptive nature of the approach is enabled by uncertainty sampling strategy to automatic selection of samples to be presented to human experts for annotation.

We investigate the memorability of a 5-season span of a popular crime-drama TV series, CSI, through the application of a vision transformer fine-tuned on the task of predicting video memorability. By investigating the popular genre of crime-drama TV through the use of a detailed annotated corpus combined with video memorability scores, we show how to extrapolate meaning from the memorability scores generated on video shots. We perform a quantitative analysis to relate video shot memorability to a variety of aspects of the show. The insights we present in this paper illustrate the importance of video memorability in applications which use multimedia in areas like education, marketing, indexing, as well as in the case here namely TV and film production.

Memories are the tethering threads that tie us to the world, and memorability is the measure of their tensile strength. The threads of memory are spun from fibres of many modalities, obscuring the contribution of a single fibre to a thread's overall tensile strength. Unfurling these fibres is the key to understanding the nature of their interaction, and how we can ultimately create more meaningful media content. In this paper, we examine the influence of audio on video recognition memorability, finding evidence to suggest that it can facilitate overall video recognition memorability rich in high-level (gestalt) audio features. We introduce a novel multimodal deep learning-based late-fusion system that uses audio gestalt to estimate the influence of a given video's audio on its overall short-term recognition memorability, and selectively leverages audio features to make a prediction accordingly. We benchmark our audio gestalt based system on the Memento10k short-term video memorability dataset, achieving top-2 state-of-the-art results.

The world has become multimodal. In addition to text, we have been sharing a huge amount of multimedia information in the form of images and videos on the Internet. The wide spread use of smart mobile devices has also changed the way we interact with the Internet. It is now natural for us to capture images and videos freely and use as part of a query, in addition to the traditional text and voices. These, along with the rapid advancements in multimedia, natural language processing, information retrieval, and conversation technologies, mean that it is time for us to explore multimodal conversation and its roles in search and recommendation. Multimodal conversation has the potential to help us to uncover and digest the huge amount of multimedia information and knowledge hidden within many systems. It also enables a natural 2-way interactions between humans and machines, with mutual benefits in enriching their respective knowledge. Finally, it opens up the possibilities of disrupting many existing applications and launching new innovative applications. This panel is timely and aims to explore this emerging trend, and discuss its potential benefits and pitfalls to society. The panel will also explore the limitations of current technologies and highlight future research directions towards developing a multimedia conversational system.

Machine learning approaches for more efficient video compression have been developed thanks to breakthroughs in deep learning. However, they typically bring coding improvements at the cost of significant increases in computational complexity, making them largely unsuitable for practical applications. In this paper, we present open-source software for convolutional neural network-based solutions which improve the interpolation of reference samples needed for fractional precision motion compensation. Contrary to previous efforts, the networks are fully linear, allowing them to be interpreted, with a full interpolation filter set derived from trained models, making it simple to integrate in conventional video coding schemes. When implemented in the context of the state-of-the-art Versatile Video Coding (VVC) test model, the complexity of the learned interpolation schemes is significantly reduced compared to the interpolation with full neural networks, while achieving notable coding efficiency improvements on lower resolution video sequences. The open-source software package is available at https://github.com/bbc/cnn-fractional-motion-compensation under the 3-clause BSD license.

Neural networks can be used in video coding to improve chroma intra-prediction. In particular, usage of fully-connected networks has enabled better cross-component prediction with respect to traditional linear models. Nonetheless, state-of-the-art architectures tend to disregard the location of individual reference samples in the prediction process. This paper proposes a new neural network architecture for cross-component intra-prediction. The network uses a novel attention module to model spatial relations between reference and predicted samples. The proposed approach is integrated into the Versatile Video Coding (VVC) prediction pipeline. Experimental results demonstrate compression gains over the latest VVC anchor compared with state-of-the-art chroma intra-prediction methods based on neural networks.

Deep learning has shown great potential in image and video compression tasks. However, it brings bit savings at the cost of significant increases in coding complexity, which limits its potential for implementation within practical applications. In this paper, a novel neural network-based tool is presented which improves the interpolation of reference samples needed for fractional precision motion compensation. Contrary to previous efforts, the proposed approach focuses on complexity reduction achieved by interpreting the interpolation filters learned by the networks. When the approach is implemented in the Versatile Video Coding (VVC) test model, up to 4.5% BD-rate saving for individual sequences is achieved compared with the baseline VVC, while the complexity of learned interpolation is significantly reduced compared to the application of full neural network.

This work investigates the use of class-level difficulty factors in multi-label classification problems for the first time. Four class-level difficulty factors are proposed: frequency, visual variation, semantic abstraction, and class co-occurrence. Once computed for a given multi-label classification dataset, these difficulty factors are shown to have several potential applications including the prediction of class-level performance across datasets and the improvement of predictive performance through difficulty weighted optimisation. Significant improvements to mAP and AUC performance are observed for two challenging multi-label datasets (WWW Crowd and Visual Genome) with the inclusion of difficulty weighted optimisation. The proposed technique does not require any additional computational complexity during training or inference and can be extended over time with inclusion of other class-level difficulty factors.

Advanced Driver-Assistance Systems (ADAS) have been attracting attention from many researchers. Vision-based sensors are the closest way to emulate human driver visual behaviour while driving. In this paper, we explore possible ways to use visual attention (saliency) for object detection and tracking. We investigate: 1) How a visual attention map such as a subjectness attention or saliency map and an objectness attention map can facilitate region proposal generation in a 2-stage object detector; 2) How a visual attention map can be used for tracking multiple objects. We propose a neural network that can simultaneously detect objects as and generate objectness and subjectness maps to save computational power. We further exploit the visual attention map during tracking using a sequential Monte Carlo probability hypothesis density (PHD) filter. The experiments are conducted on KITTI and DETRAC datasets. The use of visual attention and hierarchical features has shown a considerable improvement of ≈8% in object detection which effectively increased tracking performance by ≈4% on KITTI dataset.

Multi-word expressions (MWEs) are a hot topic in research in natural language processing (NLP), including topics such as MWE detection, MWE decomposition, and research investigating the exploitation of MWEs in other NLP fields such as Machine Translation. However, the availability of bilingual or multi-lingual MWE corpora is very limited. The only bilingual MWE corpora that we are aware of is from the PARSEME (PARSing and Multi-word Expressions) EU Project. This is a small collection of only 871 pairs of English-German MWEs. In this paper, we present multi-lingual and bilingual MWE corpora that we have extracted from root parallel corpora. Our collections are 3,159,226 and 143,042 bilingual MWE pairs for German-English and Chinese-English respectively after filtering. We examine the quality of these extracted bilingual MWEs in MT experiments. Our initial experiments applying MWEs in MT show improved translation performances on MWE terms in qualitative analysis and better general evaluation scores in quantitative analysis, on both German-English and Chinese-English language pairs. We follow a standard experimental pipeline to create our MultiMWE corpora which are available online. Researchers can use this free corpus for their own models or use them in a knowledge base as model features.

This is a guide on how to obtain and deploy the artifact and the expected results.

Sign language is the non-verbal communication used by people with hearing and speaking impairments. The automatic recognition of sign languages is usually based on video analysis of the signer though this is difficult when considering different light levels or the surrounding environment. The work in this paper uses electromyography (EMG) and focuses on letters of the Irish Sign Language (ISL) alphabet. EMG is the recording of the electrical activity produced to stimulate movement in the skeletal muscles. We capture muscle signals and inertial movement data using the Thalmic MYO armband and, in real time, recognise the ISL alphabet. Our implementation is based on signal processing, feature extraction and machine learning. The only input required to translate the ISL gestures are EMG and movement data, thus our approach is usable in scenarios where using video for automatic recognition video is not possible.

This short paper presents what is currently the main challenge associated with using Generative Adversarial Networks (GANs) to generate visual media. That challenge is around automatically determining the authenticity of an image/video, which is of increasing importance as fake videos and images start to proliferate on social media, especially when associated with political campaigning. The paper introduces GANs, outlines how they are used to generate visual media and summarises this major challenge.

Instrumented vehicles can produce huge volumes of video data per vehicle per day that must be analysed automatically, often in real time. This analysis should include identifying the presence of objects and tagging these as semantic concepts such as car, pedestrian, etc. An important element in achieving this is the annotation of training data for machine learning algorithms, which requires accurate labels at a high-level of granularity. Current practise is to use trained human annotators who can annotate only a limited volume of video per day. In this paper, we demonstrate how a generic human saliency classifier can provide visual cues for object detection using deep learning approaches. Our work is applied to datasets for autonomous driving. Our experiments show that utilizing visual saliency improves the detection of small objects and increases the overall accuracy compared with a standalone single shot multibox detector.

Computational intelligence is making its way into a variety of popular consumer products, including wearable physiological monitors such as activity trackers and sleep trackers. Such products are very convenient for the user, but this convenience is the result of a trade-off that has ethical implications, since in almost all cases it denies the user access to their own raw data underlying the easy-to-understand analyses that the products generate for them. One problem with this is that the user is not made aware of the uncertainty of the conclusions or analyses drawn from the data; another is that it is difficult for the user to reuse his or her data in other contexts, such as to combine data from multiple sources. Even if the user did have full control of the data, this would only solve part of the problem, because most people do not have the special skills required to analyze such data. This overall problem could be solved through collaboration between the data owner and a data analysis expert, though this again introduces further problems, notably that of preserving the data owner's privacy. In this paper we analyze the aforementioned issues pertaining to the ethics of wearable intelligence, propose possible approaches to handling them, and discuss the potential social impact of the technology if the issues can be successfully overcome.

In this work, we propose a new methodology to profile individual students of computer science based on their programming design using a technique called embeddings. We investigate different approaches to analyze user source code submissions in the Python language. We compare the performances of different source code vectorization techniques to predict the correctness of a code submission. In addition, we propose a new mechanism to represent students based on their code submissions for a given set of laboratory tasks on a particular course. This way, we can make deeper recommendations for programming solutions and pathways to support student learning and progression in computer programming modules effectively at a Higher Education Institution. Recent work using Deep Learning tends to work better when more and more data is provided. However, in Learning Analytics, the number of students in a course is an unavoidable limit. Thus we cannot simply generate more data as is done in other domains such as FinTech or Social Network Analysis. Our findings indicate there is a need to learn and develop better mechanisms to extract and learn effective data features from students so as to analyze the students' progression and performance effectively.

The multimedia and multi-modal community is witnessing an explosive transformation in the recent years with major societal impact. With the unprecedented deployment of multimedia devices and systems, multimedia research is critical to our abilities and prospects in advancing state-of-the-art technologies and solving real-world challenges facing the society and the nation. To respond to these challenges and further advance the frontiers of the field of multimedia, this panel will discuss the challenges and visions that may guide future research in the next ten years.

In this work recent advances in conditional adversarial networks are investigated to develop an end-to-end architecture based on Convolutional Neural Networks (CNNs) to directly map realistic colours to an input greyscale image. Observing that existing colourisation methods sometimes exhibit a lack of colourfulness, this paper proposes a method to improve colourisation results. In particular, the method uses Generative Adversarial Neural Networks (GANs) and focuses on improvement of training stability to enable better generalisation in large multi-class image datasets. Additionally, the integration of instance and batch normalisation layers in both generator and discriminator is introduced to the popular U-Net architecture, boosting the network capabilities to generalise the style changes of the content. The method has been tested using the ILSVRC 2012 dataset, achieving improved automatic colourisation results compared to other methods based on GANs.

This Research Full Paper implements a framework that harness sources of programming learning analytics on three computer programming courses a Higher Education Institution. The platform, called PredictCS, automatically detects lower-performing or “at-risk” students in programming courses and automatically and adaptively sends them feedback. This system has been progressively adopted at the classroom level to improve personalized learning. A visual analytics dashboard is developed and accessible to Faculty. This contains information about the models deployed and insights extracted from student's data. By leveraging historical student data we built predictive models using student characteristics, prior academic history, logged interactions between students and online resources, and students' progress in programming laboratory work. Predictions were generated every week during the semester's classes. In addition, during the second half of the semester, students who opted-in received pseudo real-time personalised feedback. Notifications were personalised based on students' predicted performance on the course and included a programming suggestion from a top-student in the class if any programs submitted had failed to meet the specified criteria. As a result, this helped students who corrected their programs to learn more and reduced the gap between lower and higher-performing students.

Boliden Tara Mines Ltd. consumed 184.7 GWh of electricity in 2014, equating to over 1% of the national demand of Ireland or approximately 35,000 homes. Ireland’s industrial electricity prices, at an average of 13 c/KWh in 2014, are amongst the most expensive in Europe. Cost effective electricity procurement is ever more pressing for businesses to remain competitive. In parallel, the proliferation of intelligent devices has led to the industrial Internet of Things paradigm becoming mainstream. As more and more devices become equipped with network connectivity, smart metering is fast becoming a means of giving energy users access to a rich array of consumption data. These modern sensor networks have facilitated the development of applications to process, analyse, and react to continuous data streams in real-time. Subsequently, future procurement and consumption decisions can be informed by a highly detailed evaluation of energy usage. With these considerations in mind, this paper uses variable energy prices from Ireland’s Single Electricity Market, along with smart meter sensor data, to simulate the scheduling of an industrial-sized underground pump station in Tara Mines. The objective is to reduce the overall energy costs whilst still functioning within the system’s operational constraints. An evaluation using real-world electricity prices and detailed sensor data for 2014 demonstrates significant savings of up to 10.72% over the year compared to the existing control systems.

Periodic phenomena or oscillating signals can be found frequently in nature and recent research has observed periodicity appearing in lifelog data, the automatic digital recording of everyday activities. In this paper we are exploring periodicity and intensity of periodicity in big data settings, especially when the data is noisy, unevenly sampled and incomplete. An interesting possibility is to compute the intensity or strength of detected periodicity across the time span of a lifelog to see if it reveals changes in this strength at different times, indicating shifts in underlying behaviour. In this paper we propose several metrics to estimate the intensity of periodicity, longitudinally. Evaluation of these metrics is conducted on simulated high-level activity data generated from a proposed model. We also explore periodicity intensity calculated from two real lifelog datasets using. One is “big” data consists of low-level accelerometer data and another one is high level athletic performance data.

Among the factors known to encourage healthy ageing is routine physical activity, a behaviour that is not common among the older age group. A Persuasive System Design (PSD) model offers guidelines for designing and evaluating systems aimed at reinforcing, changing or shaping underlying human behaviour and attitudes. The objective of this study was to investigate the perceptions of older workers towards persuasive principles of PSD that was integrated into an application specifically designed to encourage physical activity. Ten older workers aged 50–64 years with different physical activity levels participated in this study. Using a think-aloud technique, the participants interacted with a physical activity application, while verbally expressing their perceptions towards the persuasive elements. The results indicated that the older worker participants had positive views towards persuasive design principles that fell under the categories of primary task, dialogue support and credibility support. However, the persuasive principle of the social support category received contradictory views. Further, it was discovered that the personalisation of persuasive principles, the credibility of tailored contents and the establishment of a sense of similarity are imperative in the designing of effective persuasive physical activity applications targeting older workers.

Learning analytics are being used in many educational applications in order to help students and Faculty. In our work we use predictive analytics, using student behaviour to predict the likely performance of end of semester final grades with a system we call PredictED. The main contribution of our approach is that our intervention automatically emailed students on a regular basis, with our prediction for the outcome of their exam performance. We targeted first year, first semester University students who often struggle with making the transition into University life where they are given much more responsibility for things like attending class, completing assignments, etc. The form of student behaviour that we used is students’ levels and types of engagement with the University’s Virtual Learning Environment (VLE), Moodle. We mined the Moodle access log files for a range of parameters based on temporal as well as content access, and use machine learning techniques to predict likely pass/fail, on a weekly basis throughout the semester using logs and outcomes from previous years as training material. We chose ten first-year modules with reasonably high failure rates, large enrolments and stability of module content across the years to implement an early warning system on. From these modules 1,558 students were registered for one of these modules. They were offered the chance to opt into receiving weekly email alerts warning them about their likely outcome. Of these 75 % or 1,181 students opted into this service. Pre-intervention there were no differences between participants and non-participants on a number of measures related to previous academic record. However, post-intervention the first-attempt final grade performance yielded nearly 3 % improvement (58.4 % to 61.2 %) on average for those who opted in. This tells us that providing weekly guidance and personalised feedback to vulnerable first year students, automatically generated from monitoring of their online behaviour, has a significant positive effect on their exam performance.

This paper explores the potential for using Brain Computer Interfaces (BCI) as a relevance feedback mechanism in content-based image retrieval. Several experiments are performed using a rapid serial visual presentation (RSVP) of images at different rates (5Hz and 10Hz) on 8 users with different degrees of familiarization with BCI and the dataset. We compare the feedback from the BCI and mouse-based interfaces in a subset of TRECVid images, finding that, when users have limited time to annotate the images, both interfaces are comparable in performance. Comparing our best users in a retrieval task, we found that EEG-based relevance feedback can outperform mouse-based feedback.

Lifelogging is the ambient, continuous digital recording of a person's everyday activities for a variety of possible applications. Much of the work to date in lifelogging has focused on developing sensors, capturing information, processing it into events and then supporting event-based access to the lifelog for applications like memory recall, behaviour analysis or similar. With the recent arrival of aggregating platforms such as Apple's HealthKit, Microsoft's HealthVault and Google's Fit, we are now able to collect and aggregate data from lifelog sensors, to centralize the management of data and in particular to search for and detect patterns of usage for individuals and across populations. In this paper, we present a framework that detects both low-level and high-level periodicity in lifelog data, detecting hidden patterns of which users would not otherwise be aware. We detect periodicities of time series using a combination of correlograms and periodograms, using various signal processing algorithms. Periodicity detection in lifelogs is particularly challenging because the lifelog data itself is not always continuous and can have gaps as users may use their lifelog devices intermittingly. To illustrate that periodicity can be detected from such data, we apply periodicity detection on three lifelog datasets with varying levels of completeness and accuracy.

This paper explores the potential of brain-computer interfaces in segmenting objects from images. Our approach is centered around designing an effective method for displaying the image parts to the users such that they generate measurable brain reactions. When an image region, specifically a block of pixels, is displayed we estimate the probability of the block containing the object of interest using a score based on EEG activity. After several such blocks are displayed, the resulting probability map is binarized and combined with the GrabCut algorithm to segment the image into object and background regions. This study shows that BCI and simple EEG analysis are useful in locating object boundaries in images.

No abstract available.

This demonstration shows the integration of video analysis and search tools to facilitate the interactive retrieval of video segments depicting specific activities from surveillance footage. The implementation was developed by members of the SAVASA project for participation in the interactive surveillance event detection (SED) task of TRECVid 2012. This year, for the first time, the purpose of the interactive SED task was to evaluate systems' ability to support users in identifying video segments that depict a specific activity (event) in a large collection of surveillance video footage. Project partners worked together to analyse video and provide a query interface enabling users to search and identify matching video segments. The collaborative integration of components from multiple partners and the participation of end user partners in evaluating the system are the novel aspects of this work.

This paper presents work on integrating multiple computer vision-based approaches to surveillance video analysis to support user retrieval of video segments showing human activities. Applied computer vision using real-world surveillance video data is an extremely challenging research problem, independently of any information retrieval (IR) issues. Here we describe the issues faced in developing both generic and specific analysis tools and how they were integrated for use in the new TRECVid interactive surveillance event detection task. We present an interaction paradigm and discuss the outcomes from face-to-face end user trials and the resulting feedback on the system from both professionals, who manage surveillance video, and computer vision or machine learning experts. We propose an information retrieval approach to finding events in surveillance video rather than solely relying on traditional annotation using specifically trained classifiers.

Concept based video retrieval often relies on imperfect and uncertain concept detectors. We propose a general ranking framework to define effective and robust ranking functions, through explicitly addressing detector uncertainty. It can cope with multiple concept-based representations per video segment and it allows the re-use of effective text retrieval functions which are defined on similar representations. The final ranking status value is a weighted combination of two components: the expected score of the possible scores, which represents the risk-neutral choice, and the scores' standard deviation, which represents the risk or opportunity that the score for the actual representation is higher. The framework consistently improves the search performance in the shot retrieval task and the segment retrieval task over several baselines in five TRECVid collections and two collections which use simulated detectors of varying performance.

In this demo paper, we describe our user-generated video search system, compromising of an iPad interface communicating with a remote server. The goal of this system is to provide an easy access to video content lacking textual annotations by clustering key frames. Moreover, the graphical user interface allows users to filter video content based on various semantic concepts.

Human memory is a dynamic system that makes accessible certain memories of events based on a hierarchy of information, arguably driven by personal significance. Not all events are remembered, but those that are tend to be more psychologically relevant. In contrast, lifelogging is the process of automatically recording aspects of one's life in digital form without loss of information. In this article we share our experiences in designing computer-based solutions to assist people review their visual lifelogs and address this contrast. The technical basis for our work is automatically segmenting visual lifelogs into events, allowing event similarity and event importance to be computed, ideas that are motivated by cognitive science considerations of how human memory works and can be assisted. Our work has been based on visual lifelogs gathered by dozens of people, some of them with collections spanning multiple years. In this review article we summarize a series of studies that have led to the development of a browser that is based on human memory systems and discuss the inherent tension in storing large amounts of data but making the most relevant material the most accessible.

The nine papers in this special section on object and event classification in large-scale video collections can be categorized into four themes: video indexing, concept detection, video summarization, and event recognition.

Abstract This paper reports on an investigation into the scholarly impact of the TRECVid (Text Retrieval and Evaluation Conference, Video Retrieval Evaluation) benchmarking conferences between 2003 and 2009. The contribution of TRECVid to research in video retrieval is assessed by analyzing publication content to show the development of techniques and approaches over time and by analyzing publication impact through publication numbers and citation analysis. Popular conference and journal venues for TRECVid publications are identified in terms of number of citations received. For a selection of participants at different career stages, the relative importance of TRECVid publications in terms of citations vis à vis their other publications is investigated. TRECVid, as an evaluation conference, provides data on which research teams ‘scored’ highly against the evaluation criteria and the relationship between ‘top scoring’ teams at TRECVid and the ‘top scoring’ papers in terms of citations is analyzed. A strong relationship was found between ‘success’ at TRECVid and ‘success’ at citations both for high scoring and low scoring teams. The implications of the study in terms of the value of TRECVid as a research activity, and the value of bibliometric analysis as a research evaluation tool, are discussed.

Microblogs as a new textual domain offer a unique proposition for sentiment analysis. Their short document length suggests any sentiment they contain is compact and explicit. However, this short length coupled with their noisy nature can pose difficulties for standard machine learning document representations. In this work we examine the hypothesis that it is easier to classify the sentiment in these short form documents than in longer form documents. Surprisingly, we find classifying sentiment in microblogs easier than in blogs and make a number of observations pertaining to the challenge of supervised learning for sentiment analysis in microblogs.

Colour is everywhere in our daily lives and impacts things like our mood, yet we rarely take notice of it. One method of capturing and analysing the predominant colours that we encounter is through visual lifelogging devices such as the SenseCam. However an issue related to these devices is the privacy concerns of capturing image level detail. Therefore in this work we demonstrate a hardware prototype wearable camera that captures only one pixel - of the dominant colour prevelant in front of the user, thus circumnavigating the privacy concerns raised in relation to lifelogging. To simulate whether the capture of dominant colour would be sufficient we report on a simulation carried out on 1.2 million SenseCam images captured by a group of 20 individuals.

In this work we discuss a new, but highly relevant, topic to the multimedia community; systems to inform individuals of their carbon footprint, which could ultimately effect change in community carbon footprint-related activities. The reduction of carbon emissions is now an important policy driver of many governments, and one of the major areas of focus is in reducing the energy demand from the consumers i.e. all of us individually.

In this paper, we present prototype interactive TV software that incorporates visual content analysis tools and social networking in the home TV. We present the challenges of working with the living room TV environment and outline how we have utilized visual processing and search technologies to address these challenges and create a novel prototype interactive TV system.

Colour permeates our daily lives, yet we rarely take notice of it. In this work we utilise the SenseCam (a visual lifelogging tool), to investigate the predominant colours in one million minutes of human life that a group of 20 individuals encounter throughout their normal daily activities. We also compare the colours that different groups of people are exposed to in their typical days. This information is presented in using a novel colour-wheel visualisation which is a new means of illustrating that people are exposed to bright colours over longer durations of time during summer months, and more dark colours during winter months.

Google Scholar allows researchers to search through a free and extensive source of information on scientific publications. In this paper we show that within the limited context of SIGIR proceedings, the rankings created by Google Scholar are both significantly different and very negatively correlated with those of domain experts.

Content-Based Multimedia Information Retrieval (CBMIR) systems which leverage multiple retrieval experts (En) often employ a weighting scheme when combining expert results through data fusion. Typically however a query will comprise multiple query images (Im) leading to potentially N × M weights to be assigned. Because of the large number of potential weights, existing approaches impose a hierarchy for data fusion, such as uniformly combining query image results from a single retrieval expert into a single list and then weighting the results of each expert. In this paper we will demonstrate that this approach is sub-optimal and leads to the poor state of CBMIR performance in benchmarking evaluations. We utilize an optimization method known as Coordinate Ascent to discover the optimal set of weights (|En| ⋅ |Im|) which demonstrates a dramatic difference between known results and the theoretical maximum. We find that imposing common combinatorial hierarchies for data fusion will half the optimal performance that can be achieved. By examining the optimal weight sets at the topic level, we observe that approximately 15% of the weights (from set |En| ⋅ |Im|) for any given query, are assigned 70%-82% of the total weight mass for that topic. Furthermore we discover that the ideal distribution of weights follows a log-normal distribution. We find that we can achieve up to 88% of the performance of fully optimized query using just these 15% of the weights. Our investigation was conducted on TRECVID evaluations 2003 to 2007 inclusive and ImageCLEFPhoto 2007, totalling 181 search topics optimized over a combined collection size of 661,213 images and 1,594 topic images.

Classical information retrieval is based around a user having an information need, formulated as a query, and a system which matches the query against 'documents', retrieving those most likely to be relevant. In some applications there are challenges because the 'documents' are not discrete objects but highly inter-connected, and IR research has for decades developed models of the processes, devised novel ranking algorithms, and developed very elaborate benchmarking techniques for performance. But what if the information we need or seek is not neatly divided into documents, either discrete or inter-connected, but needs to be taken from a constant stream of data values, namely data from sensors. These sensors cover the physical sensors around us (environment, place, physical activities like traffic, weather, people movement, crowd gatherings like concerts and sports events) as well as the online sensors we have access to (blogs, tweets, etc.). Often termed the *sensor web*, this information source is characterised as being noisy, errorsome, unpredictable and dynamic, exactly like the real and the virtual worlds in which we live, work and play. In this presentation I introduce several diverse sensor web applications to show the breadth and pervasive nature of the sensor web and I then show some of the techniques which we use to manage the information which forms part of the sensor web.

The increased online presence of jihadists has raised the possibility of individuals being radicalised via the Internet. To date, the study of violent radicalisation has focused on dedicated jihadist websites and forums. This may not be the ideal starting point for such research, as participants in these venues may be described as "already made-up minds". Crawling a global social networking platform, such as YouTube, on the other hand, has the potential to unearth content and interaction aimed at radicalisation of those with little or no apparent prior interest in violent jihadism. This research explores whether such an approach is indeed fruitful. We collected a large dataset from a group within YouTube that we identified as potentially having a radicalising agenda. We analysed this data using social network analysis and sentiment analysis tools, examining the topics discussed and what the sentiment polarity (positive or negative) is towards these topics. In particular, we focus on gender differences in this group of users, suggesting most extreme and less tolerant views among female users.

In this paper we describe how content-based analysis techniques can be used to provide much greater functionality to the users of an interactive TV (iTV) device. We describe several content-based multimedia analysis techniques and how some of these can be exploited in the iTV domain, resulting in the provision of a set of powerful functions for iTV users. To validate our ideas, we introduce an iTV application we developed which incorporates some of these techniques into a simple set of user features, in order to demonstrate the usefulness of content-based techniques for iTV. The contribution of this paper is not to provide an in-depth discussion on each of the individual content-based techniques, but rather to show how many of these powerful technologies can be incorporated into an interactive TV system.

Current mainstream audio playback paradigms do not take any account of a user's physical location or orientation in the delivery of audio through headphones or speakers. Thus audio is usually presented as a static perception whereby it is naturally a dynamic 3D phenomenon audio environment. It fails to take advantage of our innate psycho-acoustical perception that we have of sound source locations around us. Described in this paper is an operational platform which we have built to augment the sound from a generic set of wireless headphones. We do this in a way that overcomes the spatial awareness limitation of audioplay-back in indoor 3D environments which are both location-aware and sensor-equipped. This platform provides access to an audio-spatial presentation modality which by its nature lends itself to numerous cross-dissiplinary applications. In the paper we present the platform and two demonstration applications.

Outlined in this short paper is a framework for the construction of outdoor location-and direction-aware audio applications along with an example application to showcase the strengths of the framework and to demonstrate how it works. Although there has been previous work in this area which has concentrated on the spatial presentation of sound through wireless headphones, typically such sounds are presented as though originating from specific, defined spatial locations within a 3D environment. Allowing a user to move freely within this space and adjusting the sound dynamically as we do here, further enhances the perceived reality of the virtual environment. Techniques to realise this are implemented by the real-time adjustment of the presented 2 channels of audio to the headphones, using readings of the user's head orientation and location which in turn are made possible by sensors mounted upon the headphones.

As the deployment of wireless sensor networks increase and their application domain widens, the opportunity for effective use of XML filtering and streaming query engines is ever more present. XML filtering engines aim to provide efficient real-time querying of streaming XML encoded data. This paper provides a detailed analysis of several such engines, focusing on the technology involved, their capabilities, their support for XPath and their performance. Our experimental evaluation identifies which filtering engine is best suited to process a given query based on its properties. Such metrics are important in establishing the best approach to filtering XML streams on-the-fly.

Evaluation of sentiment analysis, like large-scale IR evaluation, relies on the accuracy of human assessors to create judgments. Subjectivity in judgments is a problem for relevance assessment and even more so in the case of sentiment annotations. In this study we examine the degree to which assessors agree upon sentence-level sentiment annotation. We show that inter-assessor agreement is not contingent on document length or frequency of sentiment but correlates positively with automated opinion retrieval performance. We also examine the individual annotation categories to determine which categories pose most difficulty for annotators.

The most common approach to automatic summarization and highlight detection in sports video is to train an automatic classifier to detect semantic highlights based on occurrences of low-level features such as action replays, excited commentators or changes in a scoreboard. We propose an alternative approach based on the detection of perception concepts (PCs) and the construction of Petri-Nets, which can be used for both semantic description and event detection within sports videos. Low-level algorithms to detect PCs using visual, aural and motion characteristics are proposed, and a series of Petri-Nets composed of PCs is formally defined to describe video content. We call this a perception concept network–Petri-Net (PCN–PN) model. Using PCN–PNs, personalized high-level semantic descriptions of video highlights can be facilitated and queries on high-level semantics can be achieved. A particular strength of this framework is that we can easily build semantic detectors based on PCN–PNs to search within sports videos and locate interesting events. Experimental results based on recorded sports video data across three types of sports games (soccer, basketball and rugby), and each from multiple broadcasters, are used to illustrate the potential of this framework.

Identifying the people in photos is an important need for users of photo management systems. We present MediAssist, one such system which facilitates browsing, searching and semi-automatic annotation of personal photos, using analysis of both image content and the context in which the photo is captured. This semi-automatic annotation includes annotation of the identity of people in photos. In this paper, we focus on such person annotation, and propose person identification techniques based on a combination of context and content. We propose language modelling and nearest neighbor approaches to context-based person identification, in addition to novel face color and image color content-based features (used alongside face recognition and body patch features). We conduct a comprehensive empirical study of these techniques using the real private photo collections of a number of users, and show that combining context- and content-based analysis improves performance over content or context alone.

In this paper we present preliminary work on integrating \nvisual sensing with the more traditional sensing modalities \nfor marine locations. We have deployed visual sensing at one \nof the Smart Coast WSN sites in Ireland and have built a \nsoftware platform for gathering and synchronizing all sensed \ndata. We describe how the analysis of a range of different \nsensor modalities can reinforce readings from a given noisy, \nunreliable sensor.

Automated pedestrian detection, counting, and tracking have received significant attention in the computer vision community of late. As such, a variety of techniques have been investigated using both traditional 2-D computer vision techniques and, more recently, 3-D stereo information. However, to date, a quantitative assessment of the performance of stereo-based pedestrian detection has been problematic, mainly due to the lack of standard stereo-based test data and an agreed methodology for carrying out the evaluation. This has forced researchers into making subjective comparisons between competing approaches. In this paper, we propose a framework for the quantitative evaluation of a short-baseline stereo-based pedestrian detection system. We provide freely available synthetic and real-world test data and recommend a set of evaluation metrics. This allows researchers to benchmark systems, not only with respect to other stereo-based approaches, but also with more traditional 2-D approaches. In order to illustrate its usefulness, we demonstrate the application of this framework to evaluate our own recently proposed technique for pedestrian detection and tracking.

Traditional methods for creating classifiers have two main disadvantages. Firstly, it is time consuming to acquire, or manually annotate, the training collection. Secondly, the data on which the classifier is trained may be over-generalised or too specific. This paper presents our investigations into overcoming both of these drawbacks simultaneously, by providing example applications where two data sources train each other. This removes both the need for supervised annotation or feedback, and allows rapid adaptation of the classifier to different data. Two applications are presented: one using thermal infrared and visual imagery to robustly learn changing skin models, and another using changes in saturation and luminance to learn shadow appearance parameters.

Spatiograms were introduced as a generalisation of the commonly used histogram, providing the flexibility of adding spatial context information to the feature distribution information of a histogram. The originally proposed spatiogram comparison measure has significant disadvantages that we detail here. We propose an improved measure based on deriving the Bhattacharyya coefficient for an infinite number of spatial-feature bins. Its advantages over the previous measure and over histogram-based matching are demonstrated in object tracking scenarios.

In automatic video content analysis domain, the key challenges are how to recognize important objects and how to model the spatiotemporal relationships between them. In this paper we propose a semantic content analysis model based on Perception Concepts (PCs) and Finite State Machines (FSMs) to automatically describe and detect significant semantic content within sports video. PCs are defined to represent important semantic patterns for sports videos based on identifiable feature elements. PC-FSM models are designed to describe spatiotemporal relationships between PCs. And graph matching method is used to detect high-level semantic automatically. A particular strength of this approach is that users are able to design their own highlights and transfer the detection problem into a graph matching problem. Experimental results are used to illustrate the potential of this approach.

The recent development of large-scale multimedia concept ontologies has provided a new momentum for research in the semantic analysis of multimedia repositories. Different methods for generic concept detection have been extensively studied, but the question of how to exploit the structure of a multimedia ontology and existing inter-concept relations has not received similar attention. In this paper, we present a clustering-based method for modeling semantic concepts on low-level feature spaces and study the evaluation of the quality of such models with entropy-based methods. We cover a variety of methods for assessing the similarity of different concepts in a multimedia ontology. We study three ontologies and apply the proposed techniques in experiments involving the visual and semantic similarities, manual annotation of video, and concept detection. The results show that modeling inter-concept relations can provide a promising resource for many different application areas in semantic multimedia processing.

Collaboration has been identified as an important aspect in information seeking. People meet to discuss and share ideas and through this interaction an information need is quite often identified. However the process of resolving this information need, through interacting with a search engine and performing a search task, is still an individual activity. We propose an environment which allows users to collaborate to satisfy a shared information need. We discuss ways to divide the search task amongst collaborators and propose the use of relevance feedback, a common information retrieval process, to enable the transfer of knowledge across collaborators during a search session. We describe the process by which co-searchers can collaborate effectively with little redundancy and how we can combine relevance judgements from multiple searchers into a coherent model for synchronous collaborative information retrieval

In this paper, we evaluate the appearance tracking performance of multiple fusion schemes that combine information from standard CCTV and thermal infrared spectrum video for the tracking of surveillance objects, such as people, faces, bicycles and vehicles. We show results on numerous real world multimodal surveillance sequences, tracking challenging objects whose appearance changes rapidly. Based on these results we can determine the most promising fusion schemes

This paper describes a system for object segmentation and feature extraction for surveillance video. Segmentation is performed by a dynamic vision system that fuses information from thermal infrared video with standard CCTV video in order to detect and track objects. Separate background modelling in each modality and dynamic mutual information based thresholding are used to provide initial foreground candidates for tracking. The belief in the validity of these candidates is ascertained using knowledge of foreground pixels and temporal linking of candidates. The transferable belief model is used to combine these sources of information and segment objects. Extracted objects are subsequently tracked using adaptive thermo-visual appearance models. In order to facilitate search and classification of objects in large archives, retrieval features from both modalities are extracted for tracked objects. Overall system performance is demonstrated in a simple retrieval scenario.

In this paper we present a clustering-based method for representing semantic concepts on multimodal low-level feature spaces and study the evaluation of the goodness of such models with entropy-based methods. As different semantic concepts in video are most accurately represented with different features and modalities, we utilize the relative model-wise confidence values of the feature extraction techniques in weighting them automatically. The method also provides a natural way of measuring the similarity of different concepts in a multimedia lexicon. The experiments of the paper are conducted using the development set of the TRECVID 2005 corpus together with a common annotation for 39 semantic concepts

The Fischlar-TRECVid-2004 system was developed for Dublin City University's participation in the 2004 TRECVid video information retrieval benchmarking activity. The system allows search and retrieval of video shots from over 60 hours of content. The shot retrieval engine employed is based on a combination of query text matched against spoken dialogue combined with image-image matching where a still image (sourced externally), or a keyframe (from within the video archive itself), is matched against all keyframes in the video archive. Three separate text retrieval engines are employed for closed caption text, automatic speech recognition and video OCR. Visual shot matching is primarily based on MPEG-7 low-level descriptors. The system supports relevance feedback at the shot level enabling augmentation and refinement using relevant shots located by the user. Two variants of the system were developed, one that supports both text- and image-based searching and one that supports image only search. A user evaluation experiment compared the use of the two systems. Results show that while the system combining text- and image-based searching achieves greater retrieval effectiveness, users make more varied and extensive queries with the image only based searching version.

In this paper we present a technique for supporting mobile access to digital video archives without requiring explicit user queries. The idea is to infer the interests and needs of users from their WWW browsing history and represent those needs as persistent queries to the archive. An experiment, which we present here, suggests that this technique is effective for recommending video content to users on mobile devices. We also describe how to apply these findings to a mobile interface for a digital video archive.

Sensor networks are becoming increasingly important for a wide variety of applications including environmental monitoring, building safety and emergency relief services. A typical sensor network consists of a large number of small, low-power, low-cost nodes that form a self-organized network using wireless peer-to-peer communication. Because sensor networks pose unique constraints on their operation, traditional security techniques used by conventional networks cannot be applied. In this paper we consider the operational issues and security threats to sensor networks. We discuss the state of the art in terms of sensor network security and we examine the practicality of using efficient elliptic curve algorithms and identity based encryption to deploy a secure sensor network infrastructure. We evaluate the potential for realizing this on low-power, long-life devices by measuring power consumption of the operations needed for key management in a sensor network and thus provide further evidencefor the feasibility of the approach.

Technological developments in content-based analysis of digital video information are undergoing much progress, with ideas for fully automatic systems now being proposed and demonstrated. Yet because we do not yet have robust operational video retrieval systems that can be deployed and used, the usual HCI practise of conducting a usage study and an informed iterative system design is thus not possible. Físchlár-News is one of the first automatic, content-based broadcast news analysis and archival systems that process broadcast news video so that users can search, browse, and play it in an easy-to-use manner with a conventional web browser. The system incorporates a number of state-of-the-art research components, some of which are not yet considered mature technology, yet it has been built to be robust enough to be deployed to users who are interested in access to daily news throughout a university campus. In this article we report and discuss a user-evaluation study conducted with 16 users, each of whom utilized the system freely for a one month period. Results from a detailed qualitative analysis are presented, looking at collected questionnaires, incident diaries, and interaction-log data. The findings suggest that our users employed the system in conjunction with their other news update methods, such as watching TV news at home and browsing online news websites at their workplace, their major concerns being up-to-dateness and coverage of the news content. They tried to accommodate the system to fit their established web browsing habits, and they found local news content and the ability to play self-contained news stories on their desktop as major values of the system. Our study also resulted in a detailed wishlist of new features which will help in the further development of both our and others' systems.

There are several different approaches to video retrieval which vary in sophistication, and in the level of their deployment. Some are well-known, others are not yet within our reach for any kind of large volumes of video. In particular, object-based video retrieval, where an object from within a video is used for retrieval, is often particularly desirable from a searcher's perspective. In this paper we introduce Fischlar-Simpsons, a system providing retrieval from an archive of video using any combination of text searching, keyframe image matching, shot-level browsing, as well as object-based retrieval. The system is driven by user feedback and interaction rather than having the conventional search/browse/search metaphor and the purpose of the system is to explore how users can use detected objects in a shot as part of a retrieval task.

Handheld computing devices are becoming highly connected devices with high capacity storage. This has resulted in their being able to support storage of, and access to, personal photo archives. However the only means for mobile device users to browse such archives is typically a simple one-by-one scroll through image thumbnails in the order that they were taken, or by manually organising them based on folders. In this paper we describe a system for context-based browsing of personal digital photo archives. Photos are labeled with the GPS location and time they are taken and this is used to derive other context-based metadata such as weather conditions and daylight conditions. We present our prototype system for mobile digital photo retrieval, and an experimental evaluation illustrating the utility of location information for effective personal photo retrieval.

In recent years digital cameras have seen an enormous rise in popularity, leading to a huge increase in the quantity of digital photos being taken. This brings with it the challenge of organising these large collections. We preset work which organises personal digital photo collections based on date/time and GPS location, which we believe will become a key organisational methodology over the next few years as consumer digital cameras evolve to incorporate GPS and as cameras in mobile phones spread further. The accompanying video illustrates the results of our research into digital photo management tools which contains a series of screen and user interactions highlighting how a user utilises the tools we are developing to manage a personal archive of digital photos.

The I-SPY meta-search engine uses a technique called collaborative Web search to leverage the past search behaviour (queries and selections) of a community of users in order to promote search results that are relevant to the community. In this paper we describe recent studies to clarify the benefits of this approach in situations when the behaviour of users cannot be relied upon in terms of their ability to consistently select relevant results during search sessions.

In this poster we describe alternative inverted index structures that reduce the time required to process queries, produce a higher query throughput and still return high quality results to the end user. We give results based upon the TREC Terabyte dataset showing improvements that these indices give in terms of effectiveness and efficiency.

The paper presents an approach to segmenting broadcast TV news programmes automatically into individual news stories. We first segment the programme into individual shots, and then a number of analysis tools are run on the programme to extract features to represent each shot. The results of these feature extraction tools are then combined using a support vector machine trained to detect anchorperson shots. A news broadcast can then be segmented into individual stories based on the location of the anchorperson shots within the programme. We use one generic system to segment programmes from two different broadcasters, illustrating the robustness of our feature extraction process to the production styles of different broadcasters.

In this paper we give an outline of the Fischlar system developed to enable participation in the interactive search task within TRECVID 2003. TRECVID is an annual benchmarking exercise which measures the effectiveness of various video information retrieval tasks, including interactive retrieval. The accompanying video provides a usage scenario for our TRECVID2003 system which highlights how a user uses the system in order to perform retrieval of video shots.

TRECVID is an annual exercise which encourages research in information retrieval from digital video by providing a large video test collection, uniform scoring procedures, and a forum for organizations interested in comparing their results. TRECVID benchmarking covers both interactive and manual searching by end users, as well as the benchmarking of some supporting technologies including shot boundary detection, extraction of some semantic features, and the automatic segmentation of TV news broadcasts into non-overlapping news stories. TRECVID has a broad range of over 40 participating groups from across the world and as it is now (2004) in its 4th annual cycle it is opportune to stand back and look at the lessons we have learned from the cumulative activity. In this paper we shall present a brief and high-level overview of the TRECVID activity covering the data, the benchmarked tasks, the overall results obtained by groups to date and an overview of the approaches taken by selective groups in some tasks. While progress from one year to the next cannot be measured directly because of the changing nature of the video data we have been using, we shall present a summary of the lessons we have learned from TRECVID and include some pointers on what we feel are the most important of these lessons.

In this poster we present an overview of the techniques we used to develop and evaluate a text categorisation system to automatically classify racist texts. Detecting racism is difficult because the presence of indicator words is insufficient to indicate racist texts, unlike some other text classification tasks. Support Vector Machines (SVM) are used to automatically categorise web pages based on whether or not they are racist. Different interpretations of what constitutes a term are taken, and in this poster we look at three representations of a web page within an SVM -- bag-of-words, bigrams and part-of-speech tags.

In this paper, we describe an approach to combining text and visual features from MPEG-7 descriptions of video. A video retrieval process is aligned to a text retrieval process based on the TF*IDF vector space model via clustering of low-level visual features. Our assumption is that shots within the same cluster are not only similar visually but also semantically, to a certain extent. Our experiments on the TRECVID2002 and TRECVID2003 collections show that adding extra meaning to a shot based on the shots from the same cluster is useful when each video in a collection contains a high proportion of similar shots, for example in documentaries.

This paper describes a demonstration system which automatically indexes broadcast television content for subsequent non-linear browsing. User-specified television programmes are captured in MPEG-1 format and analysed using a number of video indexing tools such as shot boundary detection, keyframe extraction, shot clustering and news story segmentation. A number of different interfaces have been developed which allow a user to browse the visual index created by these analysis tools. These interfaces are designed to facilitate users locating video content of particular interest. Once such content is located, the MPEG-1 bitstream can be streamed to the user in real-time. This paper describes both the high-level functionality of the system and the low-level indexing tools employed, as well as giving an overview of the different browsing mechanisms employed.

This paper presents an approach to segmenting individual news stories in broadcast news programmes. The approach first performs shot boundary detection and keyframe extraction on the programme. Shots are then clustered into groups based on their colour and temporal similarity. The clustering process is controlled using the groups' statistics. After clustering, a set of criteria are applied and groups are successively eliminated in order to converge upon a set of anchorperson groups. The temporal locations of the shots in these anchorperson groups are then used to segment the programme in terms of individual news items. This work is carried out within the context of a complete video indexing, browsing and retrieval system.

Abstract The TREC benchmarking exercise for information retrieval (IR) experiments has provided a forum and an opportunity for IR researchers to evaluate the performance of their approaches to the IR task and has resulted in improvements in IR effectiveness. Typically, retrieval performance has been measured in terms of precision and recall, and comparisons between different IR approaches have been based on these measures. These measures are in turn dependent on the so‐called “pool depth” used to discover relevant documents. Whereas there is evidence to suggest that the pool depth size used for TREC evaluations adequately identifies the relevant documents in the entire test data collection, we consider how it affects the evaluations of individual systems. The data used comes from the Sixth TREC conference, TREC‐6. By fitting appropriate regression models we explore whether different pool depths confer advantages or disadvantages on different retrieval systems when they are compared. As a consequence of this model fitting, a pair of measures for each retrieval run, which are related to precision and recall, emerge. For each system, these give an extrapolation for the number of relevant documents the system would have been deemed to have retrieved if an indefinitely large pool size had been used, and also a measure of the sensitivity of each system to pool size. We concur that even on the basis of analyses of individual systems, the pool depth of 100 used by TREC is adequate.

In conventional information retrieval the task of finding users' search terms in a document is simple. When the document is not available in machine readable format, optical character recognition (OCR) can usually be performed. We have developed a technique for performing information retrieval on document images in such a manner that the accuracy has great utility. The method makes generalisations about the images of characters, then performs classification of these and agglomerates the resulting character shape codes into word tokens based on character shape coding. These are sufficiently specific in their representation of the underlying words to allow reasonable performance of retrieval. Using a collection of over 250 Mbytes of document texts and queries with known relevance assessments, we present a series of experiments to determine how various parameters in the retrieval strategy affect retrieval performance and we obtain a surprisingly good result.

Traditional approaches to information retrieval are based periments in which we varied how the query-caption similarity measure used our pre-computed word-word semantic distances.Our experiments, reported in the paper, show significant improvement for this environment over the more traditional approaches to information retrieval.

Techniques of automatic natural language processing have been under development since the earliest computing machines, and in recent years these techniques have proven to be robust, reliable and efficient enough to lead to commercial products in many areas. The applications include machine translation, natural language interfaces and the stylistic analysis of texts but NLP techniques have also been applied to other computing tasks besides these. In this paper we will examine and review recent progress in using the lexical, syntactic, semantic and discourse levels of the language analysis for tasks like automatic and semi-automatic indexing of text, text retrieval, text abstracting and summarisation, thesaurus generation from text corpus and conceptual information retrieval. Our own work on the application of syntactic analysis to the matching and ranking of phrases using structured representations of texts, will be included in the overview. Finally, the prospects for gains in terms of overall retrieval effectiveness or quality will be discussed.

Traditional information has relied on the extensive use of statistical parameters in the implementation of retrieval strategies. This paper sets out to investigate whether linguistic processes can be used as part of a document retrieval strategy. This is done by predefining a level of syntactic analysis of user queries only, to be used as part of the retrieval process. A large series of experiments on an experimental test collection are reported which use a parser for noun phrases as part of the retrieval strategy. The results obtained from the experiments do yield improvements in the level of retrieval effectiveness and given the crude linguistic process used and the way it was used on queries and not on document texts, suggests that the approach of using linguistic processing in retrieval, is valid.

This paper deals with mechanisms for performing text retrieval which incorporate a degree of linguistic processing into the overall strategy. We have performed some experiments using parsing of text an a test collection of documents and queries to try and find out exactly if and how parsing could contribute to an overall improvement in retrieval effectiveness. Investigating this topic has led us to the definition of a retrieval strategy which incorporates parsing of query text and a more “shallow” parsing of document texts, whose retrieval effectiveness is investigated and described. Our results indicate that significant improvements in retrieval effectiveness can be obtained by incorporating such linguistic processing into an overall retrieval strategy.

This paper presents the results of an experimental investigation into the effects that some forms of query expansion by term addition or term deletion, have on the retrieval effectiveness of a document retrieval system. The overall search strategy used by a user is an iterative process whereby a set of user-judged relevant documents at any point in the search is used to refine and improve on the remainder of the user's search. At some point during the search, the set of relevant documents found so far can be used to modify the original query, either by the addition or deletion of search terms. This process is called ‘query modification’ or ‘query expansion’. A number of different types of query modification strategies are tried and the results obtained are presented and analysed.

article Free Access Share on The nearest neighbour problem in information retrieval: an algorithm using upperbounds Authors: A. F. Smeaton University College Dublin, Belfield, Dublin University College Dublin, Belfield, DublinView Profile , C. J. van Rijsbergen University College Dublin, Belfield, Dublin University College Dublin, Belfield, DublinView Profile Authors Info & Claims ACM SIGIR ForumVolume 16Issue 1Summer 1981 pp 83–87https://doi.org/10.1145/1013228.511767Published:31 May 1981Publication History 45citation583DownloadsMetricsTotal Citations45Total Downloads583Last 12 Months54Last 6 weeks3 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF