Retrieval-Augmented Generation (RAG) has become a foundational paradigm for integrating AI agents with external knowledge. However, current RAG models remain largely constrained by static retrieval pipelines and limited capacity for adaptive reasoning over hierarchical knowledge structures. As AI agents increasingly operate in dynamic, information-rich environments, there is a growing need for models that can reason across modalities while continuously evolving their knowledge representations. We introduce HFlow, a self-evolving multimodal agentic RAG framework grounded in hypergraph representations and hyperbolic flow-based reasoning. In our formulation, heterogeneous modalities, including text, images, audio, and structured data, are modelled as nodes within a multimodal hypergraph, while hyperedges capture higher-order semantic and cross-modal relationships. By embedding this structure in hyperbolic space, the framework preserves hierarchical and compositional knowledge. This work proposes a shift from static retrieval to continuous knowledge navigation, where reasoning emerges through geometry-aware traversal of multimodal knowledge manifolds. The proposed framework unifies retrieval, reasoning, and adaptation within a single agentic architecture, offering a new direction for scalable, context-aware AI models. We discuss early empirical evidence demonstrating improved robustness and reasoning flexibility compared to conventional Euclidean and pipeline-based multimodal RAG approaches and outline future opportunities for self-improving knowledge agents.

Research on Information Retrieval (IR) has historically prioritised high-resource languages such as English and Chinese, with less attention given to many low-resource languages. For example, Shona, a Bantu language spoken by approximately 12 million people in Zimbabwe and neighbouring countries, remains under-explored in IR research despite its widespread societal use in Southern Africa. In this work, we present a preliminary study of Shona IR using sparse and dense retrieval models, demonstrating significant performance limitations due to morphological complexity and data scarcity. Based on these findings, we propose to develop a framework to advance Shona IR by developing a large-scale benchmark dataset to support morphology-aware retrieval. We hypothesise that improving Shona IR supports equitable access to digital information and enables language-inclusive AI technologies aligned with global development priorities such as accessibility to education, dissemination of healthcare information, and digital inclusion.

Brain tumors are formed when abnormal cells grow within the brain or its surrounding tissues. Approximately 400 people in Ireland receive a primary brain tumor diagnosis each year. In the US, this number increases to almost 90,000 individuals diagnosed each year. Timely diagnosis of brain tumor is essential to saving lives and significantly reducing treatment costs. To automate this process, different Artificial Intelligence (AI) techniques have been adopted to identify brain tumors in humans. Specifically, various deep learning algorithms have been used to segment and classify brain tumors. In this paper, a systematic review is conducted based on Kitchenham & Charters methodology. We selected seven research questions to identify commonly used methods, datasets, features, metrics, and Explainable AI (XAI) approaches for AI-based analysis of brain tumors. This process starts by sourcing papers that address these techniques via the IEEE Xplore and ACM biblographic databases between January 2013 and December 2024. The papers are then filtered using specifically designed inclusion and exclusion criteria. Out of 3950 papers sourced from two electronic databases, only 101 papers were selected for this review. In summary, despite a focus on segmentation and classification, our findings indicate that no AI methods have been fully adopted in clinical practice. Furthermore, none of the reviewed papers address the specific problem of weakly-supervised brain tumor segmentation, highlighting a clear research gap in the existing literature that warrants further investigation. Also, only four articles on XAI were identified. Given the importance of transparency in network predictions for brain tumor analyses, this fact supports the need for more research in this domain.

The increasing adoption of cloud microservice architectures demands robust multivariate time series anomaly detection algorithms to ensure a reliable computing infrastructure. Although countless of these algorithms are readily available in the literature, significant concerns remain about their interpretability, particularly in cases where it is required to consider both the temporal and topological aspects of the explanation. Our proposed framework leverages attention-based spatial-temporal graph convolutional networks and ensemble learning to capture complex spatial-temporal dependencies across time series. Our approach achieves an averaged event-wise F1 score of 0.94 on two proprietary datasets. After the detection phase, an analysis of the network attention weights, at carefully selected layers, provides useful explanations of the anomalous events to cloud operators.

Recent contrastive multimodal vision-language models like CLIP have demonstrated robust open-world semantic understanding, becoming the standard image backbones for vision-language applications. However, recent findings suggest high semantic similarity between well-trained unimodal encoders, which raises a key question: Is there a plausible way to connect unimodal backbones for vision-language tasks? To this end, we propose a novel framework that aligns vision and language using frozen unimodal encoders. It involves selecting semantically similar encoders in the latent space, curating a concept-rich dataset of image-caption pairs, and training simple MLP projectors. We evaluated our approach on 12 zero-shot classification datasets and 2 image-text retrieval datasets. Our best model, utilizing DINOv2 and All-Roberta-Large text encoder, achieves 76% accuracy on ImageNet with a 20-fold reduction in data and 65-fold reduction in compute requirements compared multi-modal alignment where models are trained from scratch. The proposed framework enhances the accessibility of multimodal model development while enabling flexible adaptation across diverse scenarios. Code and curated datasets are available at github.com/mayug/freeze-align.

Current Digital Twin (DT) technology lacks the cognitive capabilities needed for true autonomy and intelligent adaptation. This paper introduces CogTwin, a hybrid cognitive architecture framework for developing Cognitive Digital Twins (CDTs). CogTwin integrates a 50ms cognitive cycle inspired by human cognition, dual knowledge graphs (static Domain Knowledge Repository (DKR) and dynamic Internal Knowledge Graph (DIKG)), a hybrid attention mechanism, and self-healing capabilities. Combining symbolic, sub-symbolic, and neuro-symbolic AI, CogTwin enables real-time learning and decision-making. Simulated smart city scenarios, including traffic incident management and power outage response, demonstrate CogTwin’s potential. Preliminary performance evaluations of the pseudocode suggest feasibility of the target 50ms cycle. The architecture also incorporates explainable AI (XAI) for transparency and human-CogTwin collaboration. CogTwin contributes towards a unified theory of cognition for DTs, laying the groundwork for more sophisticated and autonomous CDTs.

Aligned text-image encoders such as CLIP have become the de-facto model for vision-language tasks. Further-more, modality-specific encoders achieve impressive per-formances in their respective domains. This raises a cen-tral question: does an alignment exist between uni-modal vision and language encoders since they fundamentally rep-resent the same physical world? Analyzing the latent spaces structure of vision and language models on image-caption benchmarks using the Centered Kernel Alignment (CKA), we find that the representation spaces of unaligned and aligned encoders are semantically similar. In the absence of statistical similarity in aligned encoders like CLIP, we show that a possible matching of unaligned encoders exists with-out any training. We frame this as a seeded graph-matching problem exploiting the semantic similarity between graphs and propose two methods - a Fast Quadratic Assignment Problem optimization, and a novel localized CKA metric-based matching/retrieval. We demonstrate the effectiveness of this on several downstream tasks including cross-lingual, cross-domain caption matching and image classification. Code available at github.com/mayug/0-shot-llm-vision.

We present a novel approach for saliency prediction in images, leveraging parallel decoding in transformers to learn saliency solely from fixation maps. Models typically rely on continuous saliency maps, to overcome the difficulty of optimizing for the discrete fixation map. We attempt to replicate the experimental setup that generates saliency datasets. Our approach treats saliency prediction as a direct set prediction problem, via a global loss that enforces unique fixations prediction through bipartite matching and a transformer encoder-decoder architecture. By utilizing a fixed set of learned fixation queries, the cross-attention reasons over the image features to directly output the fixation points, distinguishing it from other modern saliency predictors. Our approach, named Saliency TRansformer (SalTR), achieves metric scores on par with state-of-the-art approaches on the Salicon and MIT300 benchmarks.

\beginabstract Recently, modelling heterogeneous interactions in recommender systems has attracted research interest. Real-world scenarios involve sequential multi-type user-item interactions such as ''shape view'', ''shape add-to-favourites'', ''shape add-to-cart'' and ''shape purchase''. Graph Neural Network (GNN) methods have been widely adopted in Representation Learning of similar sequential user-item interactions. Promising results have been achieved by the integration of GNNs and transformers for self-attention. However, GNN based methods suffer from limited capability in handling global user-item interaction dependencies, particularly for long sequences. Moreover, these models require high computational cost of transformers, due to the quadratic memory and time complexity with respect to sequence length. This results in memory bottlenecks and slow training especially in computational resource-constrained environments. To address these challenges, we propose the FATH model which employs Flash Attention mechanism to reduce the high-bandwidth memory usage over higher-order user-item interaction sequences. Experimental results show that our model improves the training speed and reduces the memory usage with better recommendation performance in comparison with the state-of the art baselines.

We propose an accurate and fast classification network for classification of brain tumors in MRI images that outperforms all lightweight methods investigated in terms of accuracy. We test our model on a challenging 2D T1-weighted CE-MRI dataset containing three types of brain tumors: Meningioma, Glioma and Pituitary. We introduce an l2-normalized spatial attention mechanism that acts as a regularizer against overfitting during training. We compare our results against the state-of-the-art on this dataset and show that by integrating l2-normalized spatial attention into a baseline network we achieve a performance gain of 1.79 percentage points. Even better accuracy can be attained by combining our model in an ensemble with the pretrained VGG16 at the expense of execution speed. Our code is publicly available at https://github.com/juliadietlmeier/MRI_image_classification.

Early detection of colorectal polyps is of utmost importance for their treatment and for colorectal cancer prevention. Computer vision techniques have the potential to aid professionals in the diagnosis stage, where colonoscopies are manually carried out to examine the entirety of the patient's colon. The main challenge in medical imaging is the lack of data, and a further challenge specific to polyp segmentation approaches is the difficulty of manually labeling the available data: the annotation process for segmentation tasks is very time-consuming. While most recent approaches address the data availability challenge with sophisticated techniques to better exploit the available labeled data, few of them explore the self-supervised or semi-supervised paradigm, where the amount of labeling required is greatly reduced. To address both challenges, we leverage synthetic data and propose an end-to-end model for polyp segmentation that integrates real and synthetic data to artificially increase the size of the datasets and aid the training when unlabeled samples are available. Concretely, our model, PI-CUT-Seg, transforms synthetic images with an image-to-image translation module and combines the resulting images with real images to train a segmentation model, where we use model predictions as pseudolabels to better leverage unlabeled samples. Additionally, we propose PL-CUT-Seg+, an improved version of the model that incorporates targeted regularization to address the domain gap between real and synthetic images. The models are evaluated on standard benchmarks for polyp segmentation and reach state-of-the-art results in the self- and semi-supervised setups.

Designing robust algorithms capable of training accurate neural networks on uncurated datasets from the web has been the subject of much research as it reduces the need for time consuming human labor. The focus of many previous research contributions has been on the detection of different types of label noise; however, this paper proposes to improve the correction accuracy of noisy samples once they have been detected. In many state-of-the-art contributions, a two phase approach is adopted where the noisy samples are detected before guessing a corrected pseudo-label in a semi-supervised fashion. The guessed pseudo-labels are then used in the supervised objective without ensuring that the label guess is likely to be correct. This can lead to confirmation bias, which reduces the noise robustness. Here we propose the pseudo-loss, a simple metric that we find to be strongly correlated with pseudo-label correctness on noisy samples. Using the pseudo-loss, we dynamically down weight under-confident pseudo-labels throughout training to avoid confirmation bias and improve the network accuracy. We additionally propose to use a confidence guided contrastive objective that learns robust representation on an interpolated objective between class bound (supervised) for confidently corrected samples and unsupervised representation for under-confident label corrections. Experiments demonstrate the state-of-the-art performance of our Pseudo-Loss Selection (PLS) algorithm on a variety of benchmark datasets including curated data synthetically corrupted with in-distribution and out-of-distribution noise, and two real world web noise datasets. Our experiments are fully reproducible github.com/PaulAlbert31/PLS.

The task of Generic Event Boundary Detection (GEBD) aims to detect moments in videos that are naturally perceived by humans as generic and taxonomy-free event boundaries. Modeling the dynamically evolving temporal and spatial changes in a video makes GEBD a difficult problem to solve. Existing approaches involve very complex and sophisticated pipelines in terms of architectural design choices, hence creating a need for more straightforward and simplified approaches. In this work, we address this issue by revisiting a simple and effective self-supervised method and augment it with a differentiable motion feature learning module to tackle the spatial and temporal diversities in the GEBD task. We perform extensive experiments on the challenging Kinetics-GEBD and TAPOS datasets to demonstrate the efficacy of the proposed approach compared to the other self-supervised state-of-the-art methods. We also show that this simple self-supervised approach learns motion features without any explicit motion-specific pretext task. Our results can be reproduced on $github$.

Abstract Deep learning has shown excellent performance in analysing medical images. However, datasets are difficult to obtain due privacy issues, standardization problems, and lack of annotations. We address these problems by producing realistic synthetic images using a combination of 3D technologies and generative adversarial networks. We propose CUT‐seg, a joint training where a segmentation model and a generative model are jointly trained to produce realistic images while learning to segment polyps. We take advantage of recent one‐sided translation models because they use significantly less memory, allowing us to add a segmentation model in the training loop. CUT‐seg performs better, is computationally less expensive, and requires less real images than other memory‐intensive image translation approaches that require two stage training. Promising results are achieved on five real polyp segmentation datasets using only one real image and zero real annotations. As a part of this study we release Synth‐Colon, an entirely synthetic dataset that includes 20,000 realistic colon images and additional details about depth and 3D geometry: https://enric1994.github.io/synth-colon

Abstract This paper describes a deep reinforcement learning (DRL) approach that won Phase 1 of the Real Robot Challenge (RRC) 2021, and then extends this method to a more difficult manipulation task. The RRC consisted of using a TriFinger robot to manipulate a cube along a specified positional trajectory, but with no requirement for the cube to have any specific orientation. We used a relatively simple reward function, a combination of a goal‐based sparse reward and a distance reward, in conjunction with Hindsight Experience Replay (HER) to guide the learning of the DRL agent (Deep Deterministic Policy Gradient [DDPG]). Our approach allowed our agents to acquire dexterous robotic manipulation strategies in simulation. These strategies were then deployed on the real robot and outperformed all other competition submissions, including those using more traditional robotic control techniques, in the final evaluation stage of the RRC. Here we extend this method, by modifying the task of Phase 1 of the RRC to require the robot to maintain the cube in a particular orientation, while the cube is moved along the required positional trajectory. The requirement to also orient the cube makes the agent less able to learn the task through blind exploration due to increased problem complexity. To circumvent this issue, we make novel use of a Knowledge Transfer (KT) technique that allows the strategies learned by the agent in the original task (which was agnostic to cube orientation) to be transferred to this task (where orientation matters). KT allowed the agent to learn and perform the extended task in the simulator, which improved the average positional deviation from 0.134 to 0.02 m, and average orientation deviation from 142° to 76° during evaluation. This KT concept shows good generalization properties and could be applied to any actor‐critic learning algorithm.

In this paper we propose a new approach to assess the performance of video anomaly detection algorithms. Inspired by the COCO metrics we propose a quartile based quality assessment of video anomaly detection to have a detailed breakdown of algorithm performance. The proposed assessment divides the detection into five categories based on the measurement quartiles of the position, scale and motion magnitude of anomalies. A weighted precision is introduced in the average precision calculation such that the frame-level average precision reported in categories can be compared to each other regardless of the baseline of the precision-recall curve in every category.

Vision and language models are easily transferred to other tasks. In particular, they have been demonstrated to work well in the evaluation of automatic image captioning. This has made it possible to evaluate systems without the need for references or additional information apart from the image and the caption. However, these models do not provide a straightforward way of evaluating videos. In this paper, we propose using these models for video captioning evaluation. We explore the use of both single image-based evaluation and different methods to include data from multiple frames. Experiments demonstrate that using clustering methods to select a few frames to compute the final score gives an excellent correlation with human judgment. The bias in the human annotations can also influence the metric, so we propose filtering the human assessments to discard outliers and improve the evaluation process.

Evaluating video captioning systems is a challenging task as there are multiple factors to consider; for instance: the fluency of the caption, multiple actions happening in a single scene, and the human bias of what is considered important. Most metrics try to measure how similar the system generated captions are to a single or a set of human-annotated captions. This paper presents a new method based on a deep learning model to evaluate these systems. The model is based on BERT, which is a language model that has been shown to work well in multiple NLP tasks. The aim is for the model to learn to perform an evaluation similar to that of a human. To do so, we use a dataset that contains human evaluations of system generated captions. The dataset consists of the human judgments of the captions produces by the system participating in various years of the TRECVid video to text task. BERTHA obtain favourable results, outperforming the commonly used metrics in some setups.

In recent years, an increasing number of shared E-bikes have been rolling out rapidly in our cities. It, therefore, becomes important to understand new behaviour patterns of the cyclists in using these E-bikes as a foundation for the novel design of shared micromobility services as part of the realisation of next-generation intelligent transportation systems. In this paper, we deeply investigate the users’ behaviour of shared E-bikes in a case study by using the real-world dataset collected from the shared E-bike company, MOBY, which currently operates in Dublin, Ireland. More specifically, we look into the parking behaviours of users as we know that inappropriate parking of these bikes can not only increase the management costs of the company but also result in other users’ inconveniences, especially in situations of a battery shortage, which inevitably reduces the overall operational efficacy of these shared E-bikes. Our work has conducted analysis at both bike station and individual levels in a fully anonymous and GDPR-Compliant manner, and our results have shown that up to 12.9% of shared E-bike users did not park their bikes properly at the designated stands. Different visualisation tools have been applied to better illustrate our obtained results.

A recurring focus of the deep learning community is towards reducing the labeling effort. Data gathering and annotation using a search engine is a simple alternative to generating a fully human-annotated and human-gathered dataset. Although web crawling is very time efficient, some of the retrieved images are unavoidably noisy, i.e. incorrectly labeled. Designing robust algorithms for training on noisy data gathered from the web is an important research perspective that would render the building of datasets easier. In this paper we conduct a study to understand the type of label noise to expect when building a dataset using a search engine. We review the current limitations of state-of-the-art methods for dealing with noisy labels for image classification tasks in the case of web noise distribution. We propose a simple solution to bridge the gap with a fully clean dataset using Dynamic Softening of Out-of-distribution Samples (DSOS), which we design on corrupted versions of the CIFAR-100 dataset, and compare against state-of-the-art algorithms on the web noise perturbated MiniImageNet and Stanford datasets and on real label noise datasets: WebVision 1.0 and Clothing1M. Our work is fully reproducible https://git.io/JKGcj.

As a part of Advanced Driver Assistance Systems (ADASs), Consensus-based Speed Advisory Systems (CSAS) have been proposed to recommend a common speed to a group of vehicles for specific application purposes, such as emission control and energy management. With Vehicle-to-Vehicle (V2V), Vehicle-to-Infrastructure (V2I) technologies and advanced control theories in place, state-of-the-art CSAS can be designed to get an optimal speed in a privacy-preserving and decentralized manner. However, the current method only works for specific cost functions of vehicles, and its execution usually involves many algorithm iterations leading long convergence time. Therefore, the state-of-the-art design method is not applicable to a CSAS design which requires real-time decision making. In this article, we address the problem by introducing MPC-CSAS, a Multi-Party Computation (MPC) based design approach for privacy-preserving CSAS. Our proposed method is simple to implement and applicable to all types of cost functions of vehicles. Moreover, our simulation results show that the proposed MPC-CSAS can achieve very promising system performance in just one algorithm iteration without using extra infrastructure for a typical CSAS.

Deep neural networks trained with standard cross-entropy loss memorize noisy labels, which degrades their performance. Most research to mitigate this memorization proposes new robust classification loss functions. Conversely, we propose a Multi-Objective Interpolation Training (MOIT) approach that jointly exploits contrastive learning and classification to mutually help each other and boost performance against label noise. We show that standard supervised contrastive learning degrades in the presence of label noise and propose an interpolation training strategy to mitigate this behavior. We further propose a novel label noise detection method that exploits the robust feature representations learned via contrastive learning to estimate per-sample soft-labels whose disagreements with the original labels accurately identify noisy samples. This detection allows treating noisy samples as unlabeled and training a classifier in a semi-supervised manner to prevent noise memorization and improve representation learning. We further propose MOIT+, a refinement of MOIT by fine-tuning on detected clean samples. Hyperparameter and ablation studies verify the key components of our method. Experiments on synthetic and real-world noise benchmarks demonstrate that MOIT/MOIT+ achieves state-of-the-art results. Code is available at https://git.io/JI40X.

Self-supervised representation learning can mitigate the limitations in recognition tasks with few manually labeled data but abundant unlabeled data—a common scenario in sound event research. In this work, we explore unsupervised contrastive learning as a way to learn sound event representations. To this end, we propose to use the pretext task of contrasting differently augmented views of sound events. The views are computed primarily via mixing of training examples with unrelated backgrounds, followed by other data augmentations. We analyze the main components of our method via ablation experiments. We evaluate the learned representations using linear evaluation, and in two in-domain downstream sound event classification tasks, namely, using limited manually labeled data, and using noisy labeled data. Our results suggest that unsupervised contrastive pre-training can mitigate the impact of data scarcity and increase robustness against noisy labels.

Despite the success of self-supervised representation learning on planar data, to date it has not been studied on 360° images. In this paper, we extend recent advances in contrastive learning to learn latent representations that are sufficiently invariant to be highly effective for spherical saliency prediction as a downstream task. We argue that omni-directional images are particularly suited to such an approach due to the geometry of the data domain. To verify this hypothesis, we design an unsupervised framework that effectively maximizes the mutual information between the different views from both the equator and the poles. We show that the decoder is able to learn good quality saliency distributions from the encoder embeddings. Our model compares favorably with fully-supervised learning methods on the Salient360!, VR-EyeTracking and Sitzman datasets. This performance is achieved using an encoder that is trained in a completely unsupervised way and a relatively lightweight supervised decoder (3.8 × fewer parameters in the case of the ResNet50 encoder). We believe that this combination of supervised and unsupervised learning is an important step toward flexible formulations of human visual attention. The results can be reproduced on GitHub

Reducing the amount of labels required to train convolutional neural networks without performance degradation is key to effectively reduce human annotation efforts. We propose Reliable Label Bootstrapping (ReLaB), an unsupervised preprossessing algorithm which improves the performance of semi-supervised algorithms in extremely low supervision settings. Given a dataset with few labeled samples, we first learn meaningful self-supervised, latent features for the data. Second, a label propagation algorithm propagates the known labels on the unsupervised features, effectively labeling the full dataset in an automatic fashion. Third, we select a subset of correctly labeled (reliable) samples using a label noise detection algorithm. Finally, we train a semi-supervised algorithm on the extended subset. We show that the selection of the network architecture and the self-supervised algorithm are important factors to achieve successful label propagation and demonstrate that ReLaB substantially improves semi-supervised learning in scenarios of very limited supervision on image classification benchmarks such as CIFAR-10, CIFAR-100 and mini-ImageNet. We reach average error rates of 22.34 with 1 random labeled sample per class on CIFAR-10 and lower this error to 8.46 when the labeled sample in each class is highly representative. Our work is fully reproducible: https://github.com/PaulAlbert31/ReLaB.

In this work, we evaluate contrastive models for the task of image retrieval. We hypothesise that models that are learned to encode semantic similarity among instances via discriminative learning should perform well on the task of image retrieval, where relevancy is defined in terms of instances of the same object. Through our extensive evaluation, we find that representations from models trained using contrastive methods perform on-par with (and outperforms) a pre-trained supervised baseline trained on the ImageNet labels in retrieval tasks under various configurations. This is remarkable given that the contrastive models require no explicit supervision. Thus, we conclude that these models can be used to bootstrap base models to build more robust image retrieval engines.

Exemplar-based colourisation aims to add plausible colours to a grayscale image using the guidance of a colour reference image. Most existing methods tackle the task as a style transfer problem, using a convolutional neural network (CNN) to obtain deep representations of the content of both inputs. Stylised outputs are then obtained by computing similarities between both feature representations in order to transfer the style of the reference to the content of the target input. However, in order to gain robustness towards dissimilar references, the stylised outputs need to be refined with a second colourisation network, which significantly increases the overall system complexity. This work reformulates the existing methodology introducing a novel end-to-end colourisation network that unifies the feature matching with the colourisation process. The proposed architecture integrates attention modules at different resolutions that learn how to perform the style transfer task in an unsupervised way towards decoding realistic colour predictions. Moreover, axial attention is proposed to simplify the attention operations and to obtain a fast but robust cost-effective architecture. Experimental validations demonstrate efficiency of the proposed methodology which generates high quality and visually appealing colourisation. Furthermore, the complexity of the proposed methodology is reduced compared to the state-of-the-art methods.

Modern Perceptual Visual Quality Metrics (PVQMs) for video are generally complex and non-differentiable. This makes them difficult to use as loss functions in restoration and compression tuning. Traditional metrics such as PSNR/MSE which are differentiable remain important but do not capture perceptual visual criteria. In this paper we present a DNN which models a popular perceptual video metric VMAF. In so doing, we introduce a differentiable loss function that closely matches the behaviour of a perceptual metric. Employing degradation generated with H.265 compression, our model achieves a 4.41% RMSE in predicting VMAF. This can now be deployed as a video based loss function in video enhancement and compression tasks.

An intelligent speed advisory system can be used to recommend speed for vehicles travelling in a given road network in cities. In this paper, we extend our previous work where a distributed speed advisory system has been devised to recommend an optimal consensus speed for a fleet of Internal Combustion Engine Vehicles (ICEVs) in a highway scenario. In particular, we propose a novel optimisation framework where the exact format of each vehicle’s cost function can be implicit, and our algorithm can be used to recommend multiple consensus speeds for vehicles travelling on different lanes in an urban highway scenario. Our studies show that the proposed scheme based on an improved whale optimisation algorithm can effectively reduce CO2emission generated from ICEVs while providing different recommended speed options for groups of vehicles.

Neural networks can be used in video coding to improve chroma intra-prediction. In particular, usage of fully-connected networks has enabled better cross-component prediction with respect to traditional linear models. Nonetheless, state-of-the-art architectures tend to disregard the location of individual reference samples in the prediction process. This paper proposes a new neural network architecture for cross-component intra-prediction. The network uses a novel attention module to model spatial relations between reference and predicted samples. The proposed approach is integrated into the Versatile Video Coding (VVC) prediction pipeline. Experimental results demonstrate compression gains over the latest VVC anchor compared with state-of-the-art chroma intra-prediction methods based on neural networks.

Deep learning has shown great potential in image and video compression tasks. However, it brings bit savings at the cost of significant increases in coding complexity, which limits its potential for implementation within practical applications. In this paper, a novel neural network-based tool is presented which improves the interpolation of reference samples needed for fractional precision motion compensation. Contrary to previous efforts, the proposed approach focuses on complexity reduction achieved by interpreting the interpolation filters learned by the networks. When the approach is implemented in the Versatile Video Coding (VVC) test model, up to 4.5% BD-rate saving for individual sequences is achieved compared with the baseline VVC, while the complexity of learned interpolation is significantly reduced compared to the application of full neural network.

This work investigates the use of class-level difficulty factors in multi-label classification problems for the first time. Four class-level difficulty factors are proposed: frequency, visual variation, semantic abstraction, and class co-occurrence. Once computed for a given multi-label classification dataset, these difficulty factors are shown to have several potential applications including the prediction of class-level performance across datasets and the improvement of predictive performance through difficulty weighted optimisation. Significant improvements to mAP and AUC performance are observed for two challenging multi-label datasets (WWW Crowd and Visual Genome) with the inclusion of difficulty weighted optimisation. The proposed technique does not require any additional computational complexity during training or inference and can be extended over time with inclusion of other class-level difficulty factors.

This paper investigates the dependence of existing state-of-the-art person re-identification models on the presence and visibility of human faces. We apply a face detection and blurring algorithm to create anonymized versions of several popular person re-identification datasets including Market1501, DukeMTMC-reID, CUHK03, Viper, and Airport. Using a cross-section of existing state-of-the-art models that range in accuracy and computational efficiency, we evaluate the effect of this anonymization on re-identification performance using standard metrics. Perhaps surprisingly, the effect on mAP is very small, and accuracy is recovered by simply training on the anonymized versions of the data rather than the original data. These findings are consistent across multiple models and datasets. These results indicate that datasets can be safely anonymized by blurring faces without significantly impacting the performance of person reidentification systems, and may allow for the release of new richer re-identification datasets where previously there were privacy or data protection concerns.

Advanced Driver-Assistance Systems (ADAS) have been attracting attention from many researchers. Vision-based sensors are the closest way to emulate human driver visual behaviour while driving. In this paper, we explore possible ways to use visual attention (saliency) for object detection and tracking. We investigate: 1) How a visual attention map such as a subjectness attention or saliency map and an objectness attention map can facilitate region proposal generation in a 2-stage object detector; 2) How a visual attention map can be used for tracking multiple objects. We propose a neural network that can simultaneously detect objects as and generate objectness and subjectness maps to save computational power. We further exploit the visual attention map during tracking using a sequential Monte Carlo probability hypothesis density (PHD) filter. The experiments are conducted on KITTI and DETRAC datasets. The use of visual attention and hierarchical features has shown a considerable improvement of ≈8% in object detection which effectively increased tracking performance by ≈4% on KITTI dataset.

Noisy labels are an unavoidable consequence of labeling processes and detecting them is an important step towards preventing performance degradations in Convolutional Neural Networks. Discarding noisy labels avoids a harmful memorization, while the associated image content can still be exploited in a semi-supervised learning (SSL) setup. Clean samples are usually identified using the small loss trick, i.e. they exhibit a low loss. However, we show that different noise distributions make the application of this trick less straightforward and propose to continuously relabel all images to reveal a discriminative loss against multiple distributions. SSL is then applied twice, once to improve the clean-noisy detection and again for training the final model. We design an experimental setup based on ImageNet32/64 for better understanding the consequences of representation learning with differing label noise distributions and find that non-uniform out-of-distribution noise better resembles real-world noise and that in most cases intermediate features are not affected by label noise corruption. Experiments in CIFAR-10/100, ImageNet32/64 and WebVision (real-world noise) demonstrate that the proposed label noise Distribution Robust Pseudo-Labeling (DRPL) approach gives substantial improvements over recent state-of-the-art. Code is available at https://git.io/JJ0PV.

Semi-supervised learning, i.e. jointly learning from labeled and unlabeled samples, is an active research topic due to its key role on relaxing human supervision. In the context of image classification, recent advances to learn from unlabeled samples are mainly focused on consistency regularization methods that encourage invariant predictions for different perturbations of unlabeled samples. We, conversely, propose to learn from unlabeled data by generating soft pseudo-labels using the network predictions. We show that a naive pseudo-labeling overfits to incorrect pseudo-labels due to the so-called confirmation bias and demonstrate that mixup augmentation and setting a minimum number of labeled samples per mini-batch are effective regularization techniques for reducing it. The proposed approach achieves state-of-the-art results in CIFAR-10/100, SVHN, and Mini-ImageNet despite being much simpler than other methods. These results demonstrate that pseudo-labeling alone can outperform consistency regularization methods, while the opposite was supposed in previous work. Source code is available at https://git.io/fjQsC.

Virtual Reality (VR) popularity is increasing as it is becoming more affordable for end users. Available VR hardware includes low-end inexpensive devices like Google Cardboard and high-end ones like HTC Vive or Oculus Rift, which are more expensive headsets. Using VR as a platform for content delivery allows better user engagement than other traditional methods, as VR headsets remove external distractions. Multiuser VR applications provide shared experiences where users can communicate and interact in the same virtual space. This shared environment, however, introduces challenges regarding network performance, quality of service (QoS) and sessions privacy. This paper presents a multi-user VR application and aims to evaluate network behaviour in a number of scenarios, including real VR headsets (i.e. Oculus Rift), as well as simulated ones. This QoS analysis is important for the understanding of how many VR users can be simultaneously connected with high image quality.

This work presents a so-called Smart Site Survey (SSS) system that provides an efficient, web-based platform for virtual inspection of remote sites with absolute 3D metrics. Traditional manual surveying requires sending surveyors and specialised measuring tools to the targeted scene, which takes time and requires significant human resource, and often includes human error. The proposed system provides an automated site survey tool. Sample indoor scenes including offices, storage rooms, and laboratory are used for testing purposes, and highly precise virtual scenes are restored, with the measurement accuracy of 1%, i.e. an error ±1.5cm to a 150cm length. This is comparable or superior to existing works or commercial products.

This work presents an analysis of predicting multiple future paths of moving objects in traffic scenes by leveraging Long Short-Term Memory architectures (LSTMs) and Mixture Density Networks (MDNs) in a single-shot manner. Path prediction allows estimating the future positions of objects. This is useful in important applications such as security monitoring systems, Autonomous Driver Assistance Systems and assistive technologies. Normal approaches use observed positions (tracklets) of objects in video frames to predict their future paths as a sequence of position values. This can be treated as a time series. LSTMs have achieved good performance when dealing with time series. However, LSTMs have the limitation of only predicting a single path per tracklet. Path prediction is not a deterministic task and requires predicting with a level of uncertainty. Predicting multiple paths instead of a single one is therefore a more realistic manner of approaching this task. In this work, predicting a set of future paths with associated uncertainty was archived by combining LSTMs and MDNs. The evaluation was made on the KITTI and the CityFlow datasets on three type of objects, four prediction horizons and two different points of view (image coordinates and birds-eye view

The following topics are dealt with: learning (artificial intelligence); object detection; feature extraction; video surveillance; video signal processing; convolutional neural nets; cameras; image classification; image motion analysis; neural nets.

Instrumented and autonomous vehicles can generate very high volumes of video data per car per day all of which must be annotated at a high degree of granularity, detail, and accuracy. Manually or automatically annotating videos at this level and volume is not a trivial task. Manual annotation is slow and expensive while automatic annotation algorithms have shown significant improvement over the past few years. This demonstration presents an application of multi-object tracking, path prediction, and semantic segmentation approaches to facilitate the process of multi-object video annotation for enriched tracklet extraction. Currently, these three approaches are used to enhance the annotation task but more can and will be included in the future.

This paper presents a novel approach for segmenting moving objects in unconstrained environments using guided convolutional neural networks. This guiding process relies on foreground masks from independent algorithms (i.e. state-of-the-art algorithms) to implement an attention mechanism that incorporates the spatial location of foreground and background to compute their separated representations. Our approach initially extracts two kinds of features for each frame using colour and optical flow information. Such features are combined following a multiplicative scheme to benefit from their complementarity. These unified colour and motion features are later processed to obtain the separated foreground and background representations. Then, both independent representations are concatenated and decoded to perform foreground segmentation. Experiments conducted on the challenging DAVIS 2016 dataset demonstrate that our guided representations not only outperform non-guided, but also recent and top-performing video object segmentation algorithms.

Instrumented vehicles can produce huge volumes of video data per vehicle per day that must be analysed automatically, often in real time. This analysis should include identifying the presence of objects and tagging these as semantic concepts such as car, pedestrian, etc. An important element in achieving this is the annotation of training data for machine learning algorithms, which requires accurate labels at a high-level of granularity. Current practise is to use trained human annotators who can annotate only a limited volume of video per day. In this paper, we demonstrate how a generic human saliency classifier can provide visual cues for object detection using deep learning approaches. Our work is applied to datasets for autonomous driving. Our experiments show that utilizing visual saliency improves the detection of small objects and increases the overall accuracy compared with a standalone single shot multibox detector.

Despite being robust to small amounts of label noise, convolutional neural networks trained with stochastic gradient methods have been shown to easily fit random labels. When there are a mixture of correct and mislabelled targets, networks tend to fit the former before the latter. This suggests using a suitable two-component mixture model as an unsupervised generative model of sample loss values during training to allow online estimation of the probability that a sample is mislabelled. Specifically, we propose a beta mixture to estimate this probability and correct the loss by relying on the network prediction (the so-called bootstrapping loss). We further adapt mixup augmentation to drive our approach a step further. Experiments on CIFAR-10/100 and TinyImageNet demonstrate a robustness to label noise that substantially outperforms recent state-of-the-art. Source code is available at https://git.io/fjsvE and Appendix at https://arxiv.org/abs/1904.11238.

Managing one's health is among the most personal and most important challenges. HealthMedia can be viewed as one response from the multimedia community to rise to this challenge. There is an increasing number of research work that shows how core multimedia research is becoming an important enabler for solutions with applications and relevance for the societal questions of health. Within this workshop, we continue to explore the relevance, contribution and future directions of multimedia to health care and personal health. This workshop brings together researchers from diverse topics such as multimedia, tracking, lifelogging, accessibility, HCI, but also health, medicine, and psychology to address challenges and opportunities of multimedia in and for health.

In this work recent advances in conditional adversarial networks are investigated to develop an end-to-end architecture based on Convolutional Neural Networks (CNNs) to directly map realistic colours to an input greyscale image. Observing that existing colourisation methods sometimes exhibit a lack of colourfulness, this paper proposes a method to improve colourisation results. In particular, the method uses Generative Adversarial Neural Networks (GANs) and focuses on improvement of training stability to enable better generalisation in large multi-class image datasets. Additionally, the integration of instance and batch normalisation layers in both generator and discriminator is introduced to the popular U-Net architecture, boosting the network capabilities to generalise the style changes of the content. The method has been tested using the ILSVRC 2012 dataset, achieving improved automatic colourisation results compared to other methods based on GANs.

Motion analysis is a technique used by clinicians (among many others) that quantifies human movement by using camera-based systems. Marker-based motion analysis systems have been used across a variety of application domains, from Interactive 3D Tele-Immersion (i3DTI) environments to the diagnosis of neuromuscular and musculoskeletal diseases. Although such analysis is performed in several laboratories in many countries, numerous issues exist: (1) the high cost of precise motion capture systems; (2) scarcity of qualified personnel to operate them; (3) expertise required to interpret their results; (4) space requirements to install and store these systems; (5) complexity in terms of measurement protocol required for such systems; (6) limited availability; (7) and in some situations the use of markers means they are unsuitability for certain clinical use cases (e.g. for patients recovering from orthopaedic surgery). In this paper, we present, from a system perspective, an alternative, cheaper, and more accessible system for motion analysis. The ultimate aim is to use the output of this multimodal marker-less system as part of an immersive multimedia gait re-education tool. In real-time, it will advise the user on their gait performance (as well as potentially providing accurate clinical data to clinicians). With the initial focus on the capture system, we have developed and evaluated a novel multimodal system which integrates Multiple Microsoft Kinects (which employ RGB-D cameras) with multiple Shimmer Inertial Measurement Unit (IMU) sensors. We have compared this system with the VICON system (the gold standard in motion capture). Our marker-less motion capture system combines data from 4 skeletons generating 3D and complete 360 degrees in view skeleton. The system combines unit quaternions from each Kinect joint with quaternions from 4 inertial measurement units to promote integration. We used our system to measure 3D points of 12 joints from the Kinect fused skeleton and flexion-extension angles of the knee and hip in a walking trial in 8 participants with 8--10 trials per participant. The analysis found component similarity of 0.97 for knee angles and 0.98 for hip angles. These results show that our system, through combination of Multi Kinect system and Shimmer IMUs, offers a cheaper, sufficiently accurate and more accessible human motion analysis system.

Addressing ethical issues arising from AI research, and by extension from most areas of Data Science, is a core challenge in both the academic and industry worlds. The nature of research and the specific set of technical skills involved imply that AI and Data Science researchers are not equipped to identify and anticipate such issues arising, or to establish solutions at the time a specific research project is being designed. In this paper, we discuss the need for a methodology for ethical research design that involves a broader set of skills from the start of the project. We specifically identify, from the relevant literature, a set of requirements that we argue to be needed for such a methodology. We then explore two case studies where such ethical considerations have been explored in conjunction with the development of specific research projects, in order to validate those assumptions and generalise them into a set of principles guiding an "Ethics by Design" method for conducting AI and Data Science research.

This work explores attention models to weight the contribution of local convolutional representations for the instance search task. We present a retrieval framework based on bags of local convolutional features (BLCF) that benefits from saliency weighting to build an efficient image representation. The use of human visual attention models (saliency) allows significant improvements in retrieval performance without the need to conduct region analysis or spatial verification, and without requiring any feature fine tuning. We investigate the impact of different saliency models, finding that higher performance on saliency benchmarks does not necessarily equate to improved performance when used in instance search tasks. The proposed approach outperforms the state-of-the-art on the challenging INSTRE benchmark by a large margin, and provides similar performance on the Oxford and Paris benchmarks compared to more complex methods that use off-the-shelf representations. Source code is publicly available at https:llgithub.com/imatge-upc/salbow.

In this paper we propose a technique to adapt a convolutional neural network (CNN) based object counter to additional visual domains and object types while still preserving the original counting function. Domain-specific normalisation and scaling operators are trained to allow the model to adjust to the statistical distributions of the various visual domains. The developed adaptation technique is used to produce a singular patch-based counting regressor capable of counting various object types including people, vehicles, cell nuclei and wildlife. As part of this study a challenging new cell counting dataset in the context of tissue culture and patient diagnosis is constructed. This new collection, referred to as the Dublin Cell Counting (DCC) dataset, is the first of its kind to be made available to the wider computer vision community. State-of-the-art object counting performance is achieved in both the Shanghaitech (parts A and B) and Penguins datasets while competitive performance is observed on the TRANCOS and Modified Bone Marrow (MBM) datasets, all using a shared counting model.

Evaluating image retrieval systems in a quantitative way, for example by computing measures like mean average precision, allows for objective comparisons with a ground-truth. However, in cases where ground-truth is not available, the only alternative is to collect feedback from a user. Thus, qualitative assessments become important to better understand how the system works. Visualizing the results could be, in some scenarios, the only way to evaluate the results obtained and also the only opportunity to identify that a system is failing. This necessitates developing a User Interface (UI) for a Content Based Image Retrieval (CBIR) system that allows visualization of results and improvement via capturing user relevance feedback. A well-designed UI facilitates understanding of the performance of the system, both in cases where it works well and perhaps more importantly those which highlight the need for improvement. Our open-source system implements three components to facilitate researchers to quickly develop these capabilities for their retrieval engine. We present: a web-based user interface to visualize retrieval results and collect user annotations; a server that simplifies connection with any underlying CBIR system; and a server that manages the search engine data. The software itself is described in a separate submission to the ACM MM Open Source Software Competition.

Research in multimedia and health is driven by the current technological advancements in sensors and personalized healthcare. There is an increasing amount of work that shows how core multimedia research is becoming an important enabler for solutions with applications and relevance for the societal questions of health. This workshop brings together researchers from diverse topics such as multimedia, pervasive health, lifelogging, accessibility, HCI, but also health, medicine, and psychology to address challenges and opportunities of multimedia in and for health.

In this paper, we propose a novel optical flow based features for abnormal crowd behaviour detection. The proposed feature is mainly based on the angle difference computed between the optical flow vectors in the current frame and in the previous frame at each pixel location. The angle difference information is also combined with the optical flow magnitude to produce new, effective and direction invariant event features. A one-class SVM is utilized to learn normal crowd behavior. If a test sample deviates significantly from the normal behavior, it is detected as abnormal crowd behavior. Although there are many optical flow based features for crowd behaviour analysis, this is the first time the angle difference between optical flow vectors in the current frame and in the previous frame is considered as a anomaly feature. Evaluations on UMN and PETS2009 datasets show that the proposed method performs competitive results compared to the state-of-the-art methods.

We propose a new framework for human action localization in video sequences. The option to not only detect but also localize actions in surveillance video is crucial to improving system's ability to manage high volumes of CCTV. In the approach, the action localization task is formulated the maximum-path finding problem in the directed spatio-temporal video-graph. The graph is constructed on the top of frame and temporal-based low-level features. To localize actions in the video-graph, we apply a maximum-path algorithm to find the path in the graph that is considered to be the localized action in the video. The proposed approach achieves competitive performance with the J-HMDB and the UCF-Sports dataset.

In this paper we propose ResnetCrowd, a deep residual architecture for simultaneous crowd counting, violent behaviour detection and crowd density level classification. To train and evaluate the proposed multi-objective technique, a new 100 image dataset referred to as Multi Task Crowd is constructed. This new dataset is the first computer vision dataset fully annotated for crowd counting, violent behaviour detection and density level classification. Our experiments show that a multi-task approach boosts individual task performance for all tasks and most notably for violent behaviour detection which receives a 9% boost in ROC curve AUC (Area under the curve). The trained ResnetCrowd model is also evaluated on several additional benchmarks highlighting the superior generalisation of crowd analysis models trained for multiple objectives.

Ever since the emergence of digitization, we've used the term multimedia to represent a combination of different kinds of media types, such as images, audio, and videos. As new sensing technologies emerge and are now becoming omnipresent in daily lives, the definition, role and significance of multimedia is changing. Multimedia now represents the means for communicating, cooperating, and also for monitoring numerous aspects of daily life, at various levels of granularity and application, ranging from personal to societal. With this shift, we have since moved from comprehending single media and its state toward comprehending media in terms of its use context.

This paper presents an ensemble-SVM method that features a data selection mechanism with stochastic and deterministic properties, the use of extreme value theory for classifier calibration, and the introduction of random forest for classifier combination. We applied the proposed algorithm to 2 event recognition datasets and the PASCAL2007 object detection dataset and compared it to single SVM and common computer vision ensemble-SVM methods. Our algorithm outperforms its competitors and shows a considerable boost on datasets with a limited amount of outliers.

Fast computation, efficient memory storage, and performance on par with standard state-of-the-art descriptors make binary descriptors a convenient tool for many computer vision applications. However their development is mostly tailored for static images. To respond to this limitation, we introduce TREAT (Terse Rapid Edge-Anchored Tracklets), a new binary detector and descriptor, based on tracklets. It harnesses moving edge maps to perform efficient feature detection, tracking, and description at low computational cost. Experimental results on 3 different public datasets demonstrate improved performance over other popular binary features. These experiments also provide a basis for benchmarking the performance of binary descriptors in video-based applications.

The prediction of salient areas in images has been traditionally addressed with hand-crafted features based on neuroscience principles. This paper, however, addresses the problem with a completely data-driven approach by training a convolutional neural network (convnet). The learning process is formulated as a minimization of a loss function that measures the Euclidean distance of the predicted saliency map with the provided ground truth. The recent publication of large datasets of saliency prediction has provided enough data to train end-to-end architectures that are both fast and accurate. Two designs are proposed: a shallow convnet trained from scratch, and a another deeper solution whose first three layers are adapted from another network trained for classification. To the authors' knowledge, these are the first end-to-end CNNs trained and tested for the purpose of saliency prediction.

The percentage of false alarms caused by spiders in automated surveillance can range from 20-50%. False alarms increase the workload of surveillance personnel validating the alarms and the maintenance labor cost associated with regular cleaning of webs. We propose a novel, cost effective method to detect false alarms triggered by spiders/webs in surveillance camera networks. This is accomplished by building a spider classifier intended to be a part of the surveillance video processing pipeline. The proposed method uses a feature descriptor obtained by early fusion of blur and texture. The approach is sufficiently efficient for real-time processing and yet comparable in performance with more computationally costly approaches like SIFT with bag of visual words aggregation. The proposed method can eliminate 98.5% of false alarms caused by spiders in a data set supplied by an industry partner, with a false positive rate of less than 1%.

This paper presents a new approach to crowd behaviour anomaly detection that uses a set of efficiently computed, easily interpretable, scene-level holistic features. This low-dimensional descriptor combines two features from the literature: crowd collectiveness and crowd conflict, with two newly developed crowd features: mean motion speed and a new formulation of crowd density. Two different anomaly detection approaches are investigated using these features. When only normal training data is available we use a Gaussian Mixture Model (GMM) for outlier detection. When both normal and abnormal training data is available we use a Support Vector Machine (SVM) for binary classification. We evaluate on two crowd behaviour anomaly detection datasets, achieving both state-of-the-art classification performance on the violent-flows dataset as well as better than real-time processing performance (40 frames per second).

This paper proposes a new approach to automatically quantify the severity of knee osteoarthritis (OA) from radiographs using deep convolutional neural networks (CNN). Clinically, knee OA severity is assessed using Kellgren & Lawrence (KL) grades, a five point scale. Previous work on automatically predicting KL grades from radiograph images were based on training shallow classifiers using a variety of hand engineered features. We demonstrate that classification accuracy can be significantly improved using deep convolutional neural network models pre-trained on ImageNet and fine-tuned on knee OA images. Furthermore, we argue that it is more appropriate to assess the accuracy of automatic knee OA severity predictions using a continuous distance-based evaluation metric like mean squared error than it is to use classification accuracy. This leads to the formulation of the prediction of KL grades as a regression problem and further improves accuracy. Results on a dataset of X-ray images and KL grades from the Osteoarthritis Initiative (OAI) show a sizable improvement over the current state-of-the-art.

Rapid urbanization has led to more people residing in cities than ever before, and projections estimate that 64% of the global population will be urban by 2050. Cities are beginning to explore Smart City initiatives to reduce expenses and complexities while increasing efficiency and quality of life for its citizens. To achieve this goal, advances in technology and policies are needed together with rethinking traditional solutions to transportation, safety, sustainability, among other priority areas. We propose the use of a Smart Stadium as a 'living laboratory' to identify, deploy and test Internet of Things technologies and Smart City solutions in an environment small enough to practically trial but large enough to evaluate effectiveness and scalability. The Smart Stadium for Smarter Living initiative brings together Arizona State University, Dublin City University, Intel Corporation, Gaelic Athletic Association, Sun Devil Stadium and Croke Park to explore smart environment solutions.

This work proposes a simple instance retrieval pipeline based on encoding the convolutional features of CNN using the bag of words aggregation scheme (BoW). Assigning each local array of activations in a convolutional layer to a visual word produces an assignment map, a compact representation that relates regions of an image with a visual word. We use the assignment map for fast spatial reranking, obtaining object localizations that are used for query expansion. We demonstrate the suitability of the BoW representation based on local CNN features for instance retrieval, achieving competitive performance on the Oxford and Paris buildings benchmarks. We show that our proposed system for CNN feature aggregation with BoW outperforms state-of-the-art techniques using sum pooling at a subset of the challenging TRECVid INS benchmark.

In this paper we introduce a novel platform that utilises multi-modal low-cost motion capture technology for the delivery of real-time visual feedback for sports performance. This platform supports the expansion to multi-modal interfaces that utilise haptic and audio feedback, which scales effectively with motor task complexity. We demonstrate an implementation of our platform within the field of sports performance. The platform includes low-cost motion capture through a fusion technique, combining a Microsoft Kinect V2 with two wrist inertial sensors, which make use of the accelerometer and gyroscope sensors, alongside a game-based Graphical User Interface (GUI) for instruction, visual feedback and gamified score tracking.

We propose a video graph based human action recognition framework. Given an input video sequence, we extract spatio-temporal local features and construct a video graph to incorporate appearance and motion constraints to reflect the spatio-temporal dependencies among features. them. In particular, we extend a popular dbscan density-based clustering algorithm to form an intuitive video graph. During training, we estimate a linear SVM classifier using the standard Bag-of-words method. During classification, we apply Graph-Cut optimization to find the most frequent action label in the constructed graph and assign this label to the test video sequence. The proposed approach achieves state-of-the-art performance with standard human action recognition benchmarks, namely KTH and UCF-sports datasets and competitive results for the Hollywood (HOHA) dataset.

Human motion analysis technologies have been widely employed to identify injury determining factors and provide objective and quantitative feedback to athletes to help prevent injury. However, most of these technologies are: expensive, restricted to laboratory environments, and can require significant post processing. This reduces their ecological validity, adoption and usefulness. In this paper, we present a novel wearable inertial sensor framework to accurately distinguish between symmetrical and asymmetrical running patterns in an unconstrained environment. The framework can automatically classify symmetry/asymmetry using Short Time Fourier Transform (STFT) and other time domain features in conjunction with a customized Random Forest classifier. The accuracy of the designed framework is up to 94% using 3-D accelerometer and 3-D gyroscope data from a sensor node attached on the upper back of a subject. The upper back inertial sensors data were then down-sampled by a factor of 4 to simulate utilizing low-cost inertial sensors whilst also facilitating a decrease of the computational cost to achieve near real-time application. We conclude that the proposed framework can potentially pave the way for employing low-cost sensors, such as those used in smartphones, attached on the upper back to provide injury related and performance feedback in real-time in unconstrained environments.

This paper explores novel approaches for improving the spatial codification for the pooling of local descriptors to solve the semantic segmentation problem. We propose to partition the image into three regions for each object to be described: Figure, Border and Ground. This partition aims at minimizing the influence of the image context on the object description and vice versa by introducing an intermediate zone around the object contour. Furthermore, we also propose a richer visual descriptor of the object by applying a Spatial Pyramid over the Figure region. Two novel Spatial Pyramid configurations are explored: Cartesian-based and crown-based Spatial Pyramids. We test these approaches with state-of-the-art techniques and show that they improve the Figure-Ground based pooling in the Pascal VOC 2011 and 2012 semantic segmentation challenges.

This paper explores the potential for using Brain Computer Interfaces (BCI) as a relevance feedback mechanism in content-based image retrieval. Several experiments are performed using a rapid serial visual presentation (RSVP) of images at different rates (5Hz and 10Hz) on 8 users with different degrees of familiarization with BCI and the dataset. We compare the feedback from the BCI and mouse-based interfaces in a subset of TRECVid images, finding that, when users have limited time to annotate the images, both interfaces are comparable in performance. Comparing our best users in a retrieval task, we found that EEG-based relevance feedback can outperform mouse-based feedback.

We present a demonstration of a multi-modal 3D capturing platform coupled to a motion comparison system. This work is focused on the preservation of Traditional Sports and Games, namely the Gaelic sports from Ireland and Basque sports from France and Spain. Users can learn, compare and compete in the performance of sporting gestures and compare themselves to real athletes. Our online gesture database provides a way to preserve and display a wide range of sporting gestures. The capturing devices utilised are Kinect 2 sensors and wearable inertial sensors, where the number required varies based on the requested scenario. The fusion of these two capture modalities, coupled to our inverse kinematic algorithm, allow us to synthesize a fluid and reliable 3D model of the user gestures over time. Our novel comparison algorithms provide the user with a performance score and a set of comparison curves (i.e. joint angles and angular velocities), providing a precise and valuable feedback for coaches and players.

We present a novel, low-cost, interactive, exercise-based rehabilitation system. Our research involves the investigation and development of patient-centric, sensor-based rehabilitation games and surrounding technologies. HeartHealth is designed to provide a safe, personalised and fun exercise environment that could be deployed in any exercise based rehabilitation program. HeartHealth utilises a cloud-based patient information management system built on FIWARE Generic Enablers,and motion tracking coupled with our sophisticated motion comparison algorithms. Users can record customised exercises through a doctors interface and then play the rehabilitation game where they must perform a sequence of their exercises in order to complete the game scenario. Their exercises are monitored, recorded and compared by our Motion Evaluation software and real-time feedback is than given based on the users performance.

From an early age children are often told either, you are creative you should do art but stay away from science and maths. Or that you are mathematical you should do science but you're not that creative. Compounding this there also exist some traditional barriers of artistic rhetoric that say, "don't touch, don't think and don't be creative, we've already done that for you, you can just look...". The Real Time Rolling Shutter is part of a collaborative Art/Science partnership whose core tenets are in complete contrast to this. The Art/Science exhibitions we have created have invited the public to become part of the exhibition by utilising augmented digital mirrors, Kinects, feed-back camera and projector systems and augmented reality perception helmets. The fundamental underlying principles we are trying to adhere to are to foster curiosity, intrigue, wonderment and amazement and we endeavour to draw the audience into the interactive nature of our exhibits and exclaim to everyone that you can be what ever you chose to be, and that everyone can be creative, everyone can be an artist, everyone can be a scientist... all it takes is an inquisitive mind, so come and explore the real-time rolling shutter and be creative.

Motion analysis technologies have been widely used to monitor the potential for injury and enhance athlete performance. However, most of these technologies are expensive, can only be used in laboratory environments, and examine only a few trials of each movement action. In this paper, we present a novel ambulatory motion analysis framework using wearable inertial sensors to accurately assess all of an athlete’s activities in real training environment. We first present a system that automatically classifies a large range of training activities using the discrete wavelet transform (DWT) in conjunction with a random forest classifier. The classifier is capable of successfully classifying various activities with up to 98% accuracy. Second, a computationally efficient gradient descent algorithm is used to estimate the relative orientations of the wearable inertial sensors mounted on the shank, thigh, and pelvis of a subject, from which the flexion–extension knee and hip angles are calculated. These angles, along with sacrum impact accelerations, are automatically extracted for each stride during jogging. Finally, normative data are generated and used to determine if a subject’s movement technique differed to the normative data in order to identify potential injury-related factors. For the joint angle data, this is achieved using a curve-shift registration technique. It is envisaged that the proposed framework could be utilized for accurate and automatic sports activity classification and reliable movement technique evaluation in various unconstrained environments for both injury management and performance enhancement.

Motion analysis technologies have been widely used to monitor the potential for injury and enhance athlete performance. However, most of these technologies are expensive, can only be used in laboratory environments and examine only a few trials of each movement action. In this paper, we present a novel ambulatory motion analysis framework using wearable inertial sensors to accurately assess all of an athlete's activities in an outdoor training environment. We firstly present a system that automatically classifies a large range of training activities using the Discrete Wavelet Transform (DWT) in conjunction with a Random forest classifier. The classifier is capable of successfully classifying various activities with up to 98% accuracy. Secondly, a computationally efficient gradient descent algorithm is used to estimate the relative orientations of the wearable inertial sensors mounted on the thigh and shank of a subject, from which the flexion-extension knee angle is calculated. Finally, a curve shift registration technique is applied to both generate normative data and determine if a subject's movement technique differed to the normative data in order to identify potential injury related factors. It is envisaged that the proposed framework could be utilized for accurate and automatic sports activity classification and reliable movement technique evaluation in various unconstrained environments.

REVERIE (REal and Virtual Engagement in Realistic Immersive Environments [1]) targets novel research to address the demanding challenges involved with developing state-of-the-art technologies for online human interaction. The REVERIE framework enables users to meet, socialise and share experiences online by integrating cutting-edge technologies for 3D data acquisition and processing, networking, autonomy and real-time rendering. In this paper, we describe the innovative research that is showcased through the REVERIE integrated framework through richly defined use-cases which demonstrate the validity and potential for natural interaction in a virtual immersive and safe environment. Previews of the REVERIE demo and its key research components can be viewed at www.youtube.com/user/REVERIEFP7.

This paper explores the potential of brain-computer interfaces in segmenting objects from images. Our approach is centered around designing an effective method for displaying the image parts to the users such that they generate measurable brain reactions. When an image region, specifically a block of pixels, is displayed we estimate the probability of the block containing the object of interest using a score based on EEG activity. After several such blocks are displayed, the resulting probability map is binarized and combined with the GrabCut algorithm to segment the image into object and background regions. This study shows that BCI and simple EEG analysis are useful in locating object boundaries in images.

Indoor localization based on ubiquitous WLAN has exhibited the capability of being a cheap and relatively precise technology and has been verified by many successful examples. Its performance is subject to change due to multipath propagation and changes in the environment (people, building layouts, antenna characteristics, etc.) which cannot be eliminated easily. An indoor localization and tracking algorithm is described which is based on the creation of a database of WLAN signal strengths at pre-chosen calibration points (CPs). The need for fewer CPs than in standard methods is due to use of a novel interpolation algorithm, based on the specification of robust, range and angle-dependent likelihood functions that describe the probability of a user being in the vicinity of each CP. The actual location of the user is estimated by solving a system of two non-linear equations with two unknowns derived for a pair of CPs. Different pairs of CPs can be chosen to make several estimates which then can be combined to increase the accuracy of the estimate. A variety of results are presented using challenging data collected in a typical office environment which demonstrate the accuracy that can be achieved with a reduced number of CPs.)) The method is compared to several competing localization methods and is shown to give superior results.

Sleep apnea is a common sleep disorder in which patient sleep patterns are disrupted due to recurrent pauses in breathing or by instances of abnormally low breathing. Current gold standard tests for the detection of apnea events are costly and have the addition of long waiting times. This paper investigates the use of cheap and easy to use sensors for the identification of sleep apnea events. Combinations of respiration, electrocardiography (ECG) and acceleration signals were analysed. Results show that using features, formed using the discrete wavelet transform (DWT), from the ECG and acceleration signals provided the highest classification accuracy, with an F1score of 0.914. However, the novel employment of just the accelerometer signal during classification provided a comparable F1score of 0.879. By employing one or a combination of the analysed sensors a preliminary test for sleep apnea, prior to the requirement for gold standard testing, can be performed.

We present a method that extracts effective features in videos for human action recognition. The proposed method analyses the 3D volumes along the sparse motion trajectories of a set of interest points from the video scene. To represent human actions, we generate a Bag-of-Features (BoF) model based on extracted features, and finally a support vector machine is used to classify human activities. Evaluation shows that the proposed features are discriminative and computationally efficient. Our method achieves state-of-the-art performance with the standard human action recognition benchmarks, namely KTH and Weizmann datasets.

This paper describes an extension to the graph cut interactive image segmentation algorithm based on a novel approach to addressing the well known small cut problem. The approach uses a generative contrast model to weight interaction potentials. The model attempts to capture the expected changes in color between adjacent pixels in the unlabeled area of the image using the adjacent pixels in the user interactions as training data. We compare our approach to the standard graph cuts algorithm and show that the contrast model allows a user to achieve a more accurate segmentation with fewer interactions. We additionally introduce a variant of the approach based on superpixels that further enhances performance but reduces computational complexity to ensure instant feedback for optimal user experience.

People have a natural tendency to collect things about themselves, their experiences and their shared experiences with people important to them, especially family. Similar to traditional objects such as photographs, lifelogs have been shown to support reminiscence. A lifelog is a digital archive of a person's experiences and activities and lifelog devices such as wearable cameras can automatically and continuously record events throughout a whole day. We were interested in investigating what would motivate people to lifelog. Due to the importance of shared family reminiscence between family members we focused our study on comparing shared or personal motivations with ten older and ten younger family members. We found from our results that both older and younger adults were more likely to lifelog for the purposes of information sharing and that reviewing lifelog images supported family reminiscence, reflection and story-telling. Based on these findings, recommendations are made for the design of a novel intergenerational family lifelog system.

This demonstration shows the integration of video analysis and search tools to facilitate the interactive retrieval of video segments depicting specific activities from surveillance footage. The implementation was developed by members of the SAVASA project for participation in the interactive surveillance event detection (SED) task of TRECVid 2012. This year, for the first time, the purpose of the interactive SED task was to evaluate systems' ability to support users in identifying video segments that depict a specific activity (event) in a large collection of surveillance video footage. Project partners worked together to analyse video and provide a query interface enabling users to search and identify matching video segments. The collaborative integration of components from multiple partners and the participation of end user partners in evaluating the system are the novel aspects of this work.

This paper presents work on integrating multiple computer vision-based approaches to surveillance video analysis to support user retrieval of video segments showing human activities. Applied computer vision using real-world surveillance video data is an extremely challenging research problem, independently of any information retrieval (IR) issues. Here we describe the issues faced in developing both generic and specific analysis tools and how they were integrated for use in the new TRECVid interactive surveillance event detection task. We present an interaction paradigm and discuss the outcomes from face-to-face end user trials and the resulting feedback on the system from both professionals, who manage surveillance video, and computer vision or machine learning experts. We propose an information retrieval approach to finding events in surveillance video rather than solely relying on traditional annotation using specifically trained classifiers.

We demonstrate a multimedia content information retrieval engine developed for audiovisual digital libraries targeted at media professionals. It is the first of three multimedia IR systems being developed by the AXES project. The system brings together traditional text IR and state-of-the-art content indexing and retrieval technologies to allow users to search and browse digital libraries in novel ways. Key features include: metadata and ASR search and filtering, on-the-fly visual concept classification (categories, faces, places, and logos), and similarity search (instances and faces).

A Cave Automatic Virtual Environment (CAVE) is an enclosed virtual reality room that uses multiple projectors to display images across its surfaces. It uses one or more computers to synchronise and combine the images and allows users to control virtual worlds using a host of interaction devices. Traditionally, a CAVE is used by a single user at any one time and by utilising some form of motion sensing, the user's head position can be tracked to allow for first virtual perception. The images are then displayed in stereographic 3D in order to complete the virtual reality effect. Professional CAVE installations are expensive and can cost upwards of several hundred thousand euros. This tends to act as a significant barrier to their propagation, however, as the reduction in cost of high specification computers, projectors and graphics cards continues apace, it has sparked a renewed interest in CAVE environments and given rise to the realistic possibility of setting up low cost, amateur CAVEs. Unfortunately, one of the greatest disadvantages of CAVE systems is the lack of inexpensive, easy to use, specialised software. In this paper we present an open source and easy to use CAVE software toolkit called SCReen Adjusted Panoramic Effect or SCRAPE for short. We believe that SCRAPE is the first major piece in a longer-term vision that aims to bring easy to setup, easy to use, portable CAVE systems to all types of non-expert users.

Abstract The ability to accurately achieve performance capture of athlete motion during competitive play in near real‐time promises to revolutionize not only broadcast sports graphics visualization and commentary, but also potentially performance analysis, sports medicine, fantasy sports and wagering. In this paper, we present a highly portable, non‐intrusive approach for synthesizing human athlete motion in competitive game‐play with lightweight instrumentation of both the athlete and field of play. Our data‐driven puppetry technique relies on a pre‐captured database of short segments of motion capture data to construct a motion graph augmented with interpolated motions and speed variations. An athlete's performed motion is synthesized by finding a related action sequence through the motion graph using a sparse set of measurements from the performance, acquired from both worn inertial and global location sensors. We demonstrate the efficacy of our approach in a challenging application scenario, with a high‐performance tennis athlete wearing one or more lightweight body‐worn accelerometers and a single overhead camera providing the athlete's global position and orientation data. However, the approach is flexible in both the number and variety of input sensor data used. The technique can also be adopted for searching a motion graph efficiently in linear time in alternative applications.

The ever increasing availability of high speed Internet access has led to a leap in technologies that support real-time realistic interaction between humans in online virtual environments. In the context of this work, we wish to realise the vision of an online dance studio where a dance class is to be provided by an expert dance teacher and to be delivered to online students via the web. In this paper we study some of the technical issues that need to be addressed in this challenging scenario. In particular, we describe an automatic dance analysis tool that would be used to evaluate a student's performance and provide him/her with meaningful feedback to aid improvement.

We propose a novel physics-based model for analysing team players' positions and movements on a sports playing field. The goal is to detect for each frame the region with the highest population of a given team's players and the region towards which the team is moving as they press for territorial advancement, termed the region of intent. Given the positions of team players from a plan view of the playing field at any given time, we solve a particular Poisson equation to generate a smooth distribution. The proposed distribution provides the likelihood of a point to be occupied by players so that more highly populated regions can be detected by appropriate thresholding. Computing the proposed distribution for each frame provides a sequence of distributions, which we process to detect the region of intent at any time during the game. Our model is evaluated on a field hockey dataset, and results show that the proposed approach can provide effective features that could be used to generate team statistics useful for performance evaluation or broadcasting purposes.

This demo exhibits a new application of mobile ad-hoc networks, where, a group of mobile devices are connected to allow synchronized presentation of multimedia content. The demo is in the form of an interactive tour. Participants have a mobile device and the tour is led by a guide, who takes the group on an informative tour of a locale. The tour is augmented with the presentation of multimedia content on the devices, highlighting points of interest. Content presentation is controlled by the guide and is synchronized using the ad-hoc network. This is an edutainment application but the underlying technology could be applied elsewhere including educational settings and in entertainment.

Even with today's advances in automatic indexing of multimedia content, existing coaching tools for court sports lack the ability to automatically index a competitive match into key events. This paper proposes an automatic event indexing and event retrieval system for tennis, which can be used to coach from beginners upwards. Event indexing is possible using either visual or inertial sensing, with the latter potentially providing system portability. To achieve maximum performance in event indexing, multi-sensor data integration is implemented, where data from both sensors is merged to automatically index key tennis events. A complete event retrieval system is also presented to allow coaches to build advanced queries which existing sports coaching solutions cannot facilitate without an inordinate amount of manual indexing.

Human memory is a dynamic system that makes accessible certain memories of events based on a hierarchy of information, arguably driven by personal significance. Not all events are remembered, but those that are tend to be more psychologically relevant. In contrast, lifelogging is the process of automatically recording aspects of one's life in digital form without loss of information. In this article we share our experiences in designing computer-based solutions to assist people review their visual lifelogs and address this contrast. The technical basis for our work is automatically segmenting visual lifelogs into events, allowing event similarity and event importance to be computed, ideas that are motivated by cognitive science considerations of how human memory works and can be assisted. Our work has been based on visual lifelogs gathered by dozens of people, some of them with collections spanning multiple years. In this review article we summarize a series of studies that have led to the development of a browser that is based on human memory systems and discuss the inherent tension in storing large amounts of data but making the most relevant material the most accessible.

In this paper we describe and evaluate two methods underpinning a surveillance-based content management system, designed for monitoring and profiling freight-based vehicular traffic in a seaport environment. The `InSPeCT' system captures video footage of passing vehicles, and uses tailored optical character recognition (OCR) to index the footage according to vehicle license-plates and freight codes. The system provides advanced search techniques for the efficient retrieval of records, where each captured vehicle is profiled according to captured imagery and its associated interpretations. Underpinning this system is a method for detecting the boundaries of individual transits within sustained traffic flow. Considering it desirable to extend the indexing functionality of the system beyond OCR, a colour-based vehicle indexing approach is proposed and evaluated. All evaluations take place in the context of a system deployed in a busy national seaport.

In this paper, we address the problem of reconstructing 3D volumetric models, illustrating human sporting performance for use in coaching scenarios. We advocate the use of low cost camera networks for acquiring such data, allowing the approach to be feasibly adopted by both amateur and elite level sports athletes. A dynamic voxel carving approach is described, coupled with over-head player tracking and autonomous background subtraction, to automatically produce a 3D reconstruction technique that intelligently uses memory resources. We demonstrate the efficacy of our approach in the context of tennis as a challenging application scenario.

In this paper we describe the design considerations for a touchscreen visual lifelog browser. Visual lifelogs are large collections of photographs which represent a person's experiences. Lifelogging devices, such as the wearable camera known as SenseCam, can record thousands of images per day. Utilizing the approach of event segmentation to organize and present these images, we have designed an interface to present lifelog collections for touchscreen interaction, thus increasing accessibility for users.

In this work, we describe a novel system that automatically evaluates dance performances against a gold-standard performance and provides visual feedback to the performer in a 3D virtual environment. The system acquires the motion of a performer via Kinect-based human skeleton tracking, making the approach viable for a large range of users, including home enthusiasts. Unlike traditional gaming scenarios, when the motion of a user must by kept in synch with a pre-recorded avatar that is displayed on screen, the technique described in this paper targets online interactive scenarios where dance choreographies can be set, altered, practiced and refined by users. In this work, we have addressed some areas of this application scenario. In particular, a set of appropriate signal processing and soft computing methodologies is proposed for temporally aligning dance movements from two different users and quantitatively evaluating one performance against another.

The Huawei/3DLife Grand Challenge Dataset provides multimodal recordings of Salsa dancing, consisting of audiovisual streams along with depth maps and inertial measurements. In this paper, we propose a system for augmented reality-based evaluations of Salsa dancer performances. An essential step for such a system is the automatic temporal synchronisation of the multiple modalities captured from different sensors, for which we propose efficient solutions. Furthermore, we contribute modules for the automatic analysis of dance performances and present an original software application, specifically designed for the evaluation scenario considered, which enables an enhanced dance visualisation experience, through the augmentation of the original media with the results of our automatic analyses.

Whilst voxel carving approaches exist that allow non-invasive 3D human reconstruction, their performance is heavily dependent on the number of cameras used and the placement of these cameras around the subject. We present a technique to quantify the fall-off in accuracy of spatially carved volumetric representations of humans based on real world constraints. We describe an example of such a quantitative evaluation using a synthetic dataset of typical sports motion in a tennis court scenario, created using computer graphics techniques and motion capture data. Experiments are performed using a baseline voxel carving technique that includes player tracking, background subtraction and player voxel carving. This type of quantitative evaluation could be used by amateur sporting clubs without a sophisticated capture infrastructure to understand how best to instrument a camera network in order to obtain a good trade-off between reconstruction accuracy and installation cost.

We present a demonstration of a multi-sensory virtual experience that accurately reflects the experience of a real visit to a museum. We have developed a 3D interactive digital museum exhibition corresponding to a photo-realistic 3D reconstruction of the Iconic Treasures exhibition at the National Museum of Ireland's archaeology building. Users can explore the 3D virtual environment and interact with the displays by accessing high resolution photographs and historical descriptions of the artefacts. This demo will showcase the experience on a 3D laptop with the use of active shutter LCD glasses to create an immersive experience. A key aspect of the demonstration is that it was developed using low-cost or freely available OTS tools wherever possible putting the creation of such digital exhibitions within reach of even modest budgets.

In this paper we address the automatic identification of indoor locations using a combination of WLAN and image sensing. Our motivation is the increasing prevalence of wearable cameras, some of which can also capture WLAN data. We propose to use image-based and WLAN-based localization individually and then fuse the results to obtain better performance overall. We demonstrate the effectiveness of our fusion algorithm for localisation to within a 8.9m2 room on very challenging data both for WLAN and image-based algorithms. We envisage the potential usefulness of our approach in a range of ambient assisted living applications.

This paper describes a fully-operational content-indexing and management system, designed for monitoring and profiling freight-based vehicular traffic in a seaport environment. The 'InSPeCT' system captures video footage of passing vehicles and uses tailored OCR to index the footage according to vehicle license plates and freight codes. In addition to real-time functionality such as alerting, the system provides advanced search techniques for the efficient retrieval of records, where each vehicle is profiled according to multi-angled video, context information, and links to external information sources. Currently being piloted at a busy national seaport, the feedback from port officials indicates the system to be extremely useful in supplementing their existing transportation-security structures.

In this work, a scheme for key distribution and network access in a Wireless Sensor Network (WSN) that utilizes Identity-Based Cryptography (IBC) is presented. The scheme is analyzed on the ARM920T processor and measurements were taken for the runtime and energy of its components. It was found that the Tate pairing component of the scheme consumes significant amounts of energy, and so should be ported to hardware. An accelerator was implemented in 65 nm Complementary Metal Oxide Silicon (CMOS) technology and area, timing, and energy figures have been obtained for the design. Results indicate that a hardware implementation of IBC would meet the strict energy constraint required of a wireless sensor network node.

Lifelogging is the capture and storage of everyday experiences and the act of reviewing lifelog data can significantly support episodic memory, which is particularly vulnerable to the effects of ageing. To design an accessible lifelogging application for older users we firstly need to explore what lifelogging features the application should include. We carried out a questionnaire study to investigate what lifelogging items people from different age groups are currently collecting. Also of interest to us was what items the participants would like to collect and the differences in lifelogging choices between age groups. The results from this questionnaire will contribute to the design of a lifelogging application which focuses on older users' preferences, motivations and abilities.

Breathing exercises form an essential part of the treatment for respiratory illnesses such as cystic fibrosis. Ideally these exercises should be performed on a daily basis. This paper presents an interactive system using a wearable textile sensor to monitor breathing patterns. A graphical user interface provides visual real-time feedback to patients. The aim of the system is to encourage the correct performance of prescribed breathing exercises by monitoring the rate and the depth of breathing. The system is straight forward to use, low-cost and can be installed easily within a clinical setting or in the home. Monitoring the user with a wearable sensor gives real-time feedback to the user as they perform the exercise, allowing them to perform the exercises independently. There is also potential for remote monitoring where the user's overall performance over time can be assessed by a clinician.

With the increasing volumes of home video footage and the need for effectively managing such archives, home movie summarisation has become an important and key research topic in the recent past. Despite growing interest from the research community, automatic summarisation remains a challenging research problem due to unrestricted capture and lack of storyline present in the home video content. In this paper, we address a sub-problem of the broad summarisation problem, namely sub-shot segmentation, as an important content analysis step in home movie summarisation. More specifically, we conduct an experimental study to identify an efficient and robust sub-shot segmentation method through a performance comparison of a number of optical flow field computation methods. Formulating the task of sub-shot segmentation as a process of decomposing a raw video into different video segments in line with the change in dominant camera motion, we are interested in detecting the most useful camera motions we have observed in real collections, namely pan, tilt and zoom. To this end, a number of feature-based optical flow field computation methods, including two well-known local feature descriptors called SIFT and SURF, and the Pyramidal Lucas-Kanade (PLK) local feature tracker, are investigated as potentially useful approaches for facilitating efficient and robust sub-shot segmentation in this paper. We then compare the performance of those algorithms with 5 different block matching motion estimation algorithms. Simulation results show that the PLK algorithm outperforms the SIFT and SURF algorithms in terms of both accuracy and efficiency. Furthermore, despite its slightly lower accuracy compared to that of the most optimal block matching algorithms (BMA), we conclude that a local feature tracker like the PLK algorithm is the preferred choice for sub-shot detection in real-life home movie archives due to the trade-off it provides between accuracy and efficiency.

Colour is everywhere in our daily lives and impacts things like our mood, yet we rarely take notice of it. One method of capturing and analysing the predominant colours that we encounter is through visual lifelogging devices such as the SenseCam. However an issue related to these devices is the privacy concerns of capturing image level detail. Therefore in this work we demonstrate a hardware prototype wearable camera that captures only one pixel - of the dominant colour prevelant in front of the user, thus circumnavigating the privacy concerns raised in relation to lifelogging. To simulate whether the capture of dominant colour would be sufficient we report on a simulation carried out on 1.2 million SenseCam images captured by a group of 20 individuals.

In this paper, we present prototype interactive TV software that incorporates visual content analysis tools and social networking in the home TV. We present the challenges of working with the living room TV environment and outline how we have utilized visual processing and search technologies to address these challenges and create a novel prototype interactive TV system.

Colour permeates our daily lives, yet we rarely take notice of it. In this work we utilise the SenseCam (a visual lifelogging tool), to investigate the predominant colours in one million minutes of human life that a group of 20 individuals encounter throughout their normal daily activities. We also compare the colours that different groups of people are exposed to in their typical days. This information is presented in using a novel colour-wheel visualisation which is a new means of illustrating that people are exposed to bright colours over longer durations of time during summer months, and more dark colours during winter months.

Addressing the obesity epidemic that plagues many societies remains an outstanding public health issue. One innovative approach to addressing this problem is Exergaming. A combination of "Exercise" and "Gaming", the objective is to motivate people participate in exercise regimes, usually in their home environment. In this paper a more holistic interpretation of this exercise paradigm is proposed. Freegaming augments Exergaming in a number of key dimensions but especially through the promotion of games in outdoor mobile contexts and within a social environment. The design and implementation of a platform for Freegaming is described and illustrated through the description of a sample game.

The paper reports a new fast intra prediction algorithms based on separating the transformed coefficients of neighboring blocks. The prediction blocks are obtained from the transformed and quantized neighboring blocks that generate minimum distortion for each DC and AC coefficients. To obtain fast coding with comparable coding efficiency compared to H.264/AVC, we present the full block search prediction (FBSP) and the edge based distance prediction (EBDP). These are immune to both the intra prediction error and the drift propagation; in addition, do not require a low pass filtering named extrapolation and mode decisions to obtain a prediction block. In FBSP, eight neighbor blocks (B(i-j)) are chosen to be candidates by observing the edge direction of the current block ((0,0)), where (i,j) are relative displacement position to the current block denoted as (i,j) G {(0,-1), (-1, 0), (-1, -1), (+1, -1), (-2, -1), (-1, -2), (+2, -1), (+1, -2)}, and the prediction block can be obtained by combining DC and AC coefficients of a neighboring block generating minimum distortion of residues. In EBSP, the edge directions of the current block are estimated from DCT coefficients, and intuitionally the prediction block is selected by observing the minimum edge distance (difference) compared to the current block without any distortion measurements. We have applied the proposed algorithms to MPEG test sequences which are Foreman, Hall, Mother and Daughter and Mobile. A PSNR (peak signal to noise ratio) gain of -2 to +0.8dB is observed at various bit rates shown in Figure l(a)(b), where + indicates PSNR degradation, - denotes PSNR gain compared to H.264/AVC intra coding. In case of a complex scene in terms of edges, the prediction error could be more dominant rather than one caused by quantization, the proposed method shows the same or better coding efficiency due to its insensitivity of the prediction error and the drift propagation. Analyzing of computational complexity, the proposed algorithms require less than 15% complexity compared to low complexity mode of H.264/AVC in Figure 1(c) since they does not perform an extrapolation and mode decisions which generates intensive computational complexity in H.264/AVC intra coding.

In this paper we describe how content-based analysis techniques can be used to provide much greater functionality to the users of an interactive TV (iTV) device. We describe several content-based multimedia analysis techniques and how some of these can be exploited in the iTV domain, resulting in the provision of a set of powerful functions for iTV users. To validate our ideas, we introduce an iTV application we developed which incorporates some of these techniques into a simple set of user features, in order to demonstrate the usefulness of content-based techniques for iTV. The contribution of this paper is not to provide an in-depth discussion on each of the individual content-based techniques, but rather to show how many of these powerful technologies can be incorporated into an interactive TV system.

Variable block size (VBS) based motion estimation has been adapted in state of the art video coding, such as H.264/AVC, VC-1. However, a low complexity H.264/AVC encoder cannot take advantage of VBS due to its power consumption requirements. In this paper, we present a VBS partition algorithm based on a binary motion edge map without either initial motion estimation or rate-distortion (R-D)optimization for selecting modes. The proposed algorithm uses the Walsh Hadamard transform (WHT) to create a binary edge map, which provides a computational complexity cost effectiveness compared to other light segmentation methods typically used to detect the required region.

We present an interactive and multi-level abstraction framework for user-generated video (UGV) summarisation, allowing a user the flexibility to select a summarisation criterion out of a number of methods provided by the system. First, a given raw video is segmented into shots, and each shot is further decomposed into sub-shots in line with the change in dominant camera motion. Secondly, principal component analysis (PCA) is applied to the colour representation of the collection of sub-shots, and a content map is created using the first few components. Each sub-shot is represented with a "footprint" on the content map, which reveals its content significance (coverage) and the most dynamic segment. The final stage of abstraction is devised in a user-assisted manner whereby a user is able to specify a desired summary length, with options to interactively perform abstraction at different granularity of visual comprehension. The results obtained show the potential benefit in significantly alleviating the burden of laborious user intervention associated with conventional video editing/browsing.

H.264/AVC uses variable block size motion estimation (VBS-ME) to improve coding gain. However, its complexity is significant and fixed regardless of the required quality or of the scene characteristics. In this paper, we propose an adaptive complexity algorithm based on using the Walsh Hadamard transform (WHT). VBS automatic partition and skip mode detection algorithms also are proposed. Experimental results show that 70% - 5% of the computation of H.264/AVC is required to achieve the same PSNR.

The most common approach to automatic summarization and highlight detection in sports video is to train an automatic classifier to detect semantic highlights based on occurrences of low-level features such as action replays, excited commentators or changes in a scoreboard. We propose an alternative approach based on the detection of perception concepts (PCs) and the construction of Petri-Nets, which can be used for both semantic description and event detection within sports videos. Low-level algorithms to detect PCs using visual, aural and motion characteristics are proposed, and a series of Petri-Nets composed of PCs is formally defined to describe video content. We call this a perception concept network–Petri-Net (PCN–PN) model. Using PCN–PNs, personalized high-level semantic descriptions of video highlights can be facilitated and queries on high-level semantics can be achieved. A particular strength of this framework is that we can easily build semantic detectors based on PCN–PNs to search within sports videos and locate interesting events. Experimental results based on recorded sports video data across three types of sports games (soccer, basketball and rugby), and each from multiple broadcasters, are used to illustrate the potential of this framework.

Segmentation algorithms traditionally employ low-level features to divide images into different regions that show a certain degree of homogeneity. However, low-level features, spatial or temporal, are not always reliable when processing real-world video sequences, because of issues like illuminations or complex backgrounds. Furthermore, real world objects can be composed of different regions with heterogeneous features. Although the inclusion of motion can mitigate some of these effects, many problems are still present. This paper proposes the utilization of some spatio-temporal mid-level features that are related, on the one hand, to geometric properties of real objects and, on the other, to well-known motion patterns. Specifically, the proposed algorithm uses a mid-level module that controls the subsequent segmentation using these kinds of features. Some experiments and evaluations show that the inclusion of mid-level features can help to obtain perceptually more meaningful segmentations, thus resulting in regions that are closer to semantic concepts.

In this paper we present preliminary work on integrating \nvisual sensing with the more traditional sensing modalities \nfor marine locations. We have deployed visual sensing at one \nof the Smart Coast WSN sites in Ireland and have built a \nsoftware platform for gathering and synchronizing all sensed \ndata. We describe how the analysis of a range of different \nsensor modalities can reinforce readings from a given noisy, \nunreliable sensor.

We describe in this paper a method for 3D shape indexing and retrieval that we apply on three data collections of the SHREC - SHape Retrieval Contest 2008: Stability on watertight, CAD and Generic 3D models. The method is based on a set of 2D multi-views after a pose and scale normalization of the models using Continuous PCA and the enclosing sphere. In all views we extract the models silhouettes and compare them pairwise. To compute the similitude measure we consider the external contour of the silhouettes, we extract their convexities and concavities at different scale levels and we build a multiscale representation. The pairs of contours are then compared by elastic matching achieved by using dynamic programming.

Automated pedestrian detection, counting, and tracking have received significant attention in the computer vision community of late. As such, a variety of techniques have been investigated using both traditional 2-D computer vision techniques and, more recently, 3-D stereo information. However, to date, a quantitative assessment of the performance of stereo-based pedestrian detection has been problematic, mainly due to the lack of standard stereo-based test data and an agreed methodology for carrying out the evaluation. This has forced researchers into making subjective comparisons between competing approaches. In this paper, we propose a framework for the quantitative evaluation of a short-baseline stereo-based pedestrian detection system. We provide freely available synthetic and real-world test data and recommend a set of evaluation metrics. This allows researchers to benchmark systems, not only with respect to other stereo-based approaches, but also with more traditional 2-D approaches. In order to illustrate its usefulness, we demonstrate the application of this framework to evaluate our own recently proposed technique for pedestrian detection and tracking.

Traditional methods for creating classifiers have two main disadvantages. Firstly, it is time consuming to acquire, or manually annotate, the training collection. Secondly, the data on which the classifier is trained may be over-generalised or too specific. This paper presents our investigations into overcoming both of these drawbacks simultaneously, by providing example applications where two data sources train each other. This removes both the need for supervised annotation or feedback, and allows rapid adaptation of the classifier to different data. Two applications are presented: one using thermal infrared and visual imagery to robustly learn changing skin models, and another using changes in saturation and luminance to learn shadow appearance parameters.

Spatiograms were introduced as a generalisation of the commonly used histogram, providing the flexibility of adding spatial context information to the feature distribution information of a histogram. The originally proposed spatiogram comparison measure has significant disadvantages that we detail here. We propose an improved measure based on deriving the Bhattacharyya coefficient for an infinite number of spatial-feature bins. Its advantages over the previous measure and over histogram-based matching are demonstrated in object tracking scenarios.

This paper presents a new method for segmentation of images into large regions that reflect the real world objects present in a scene. It explores the feasibility of utilizing spatial configuration of regions and their geometric properties (the so-called syntactic visual features by C. Ferran Bennstrom and JR Casas (2004)) for improving the correspondence of segmentation results produced by the well-known recursive shortest spanning tree (RSST) algorithm by O.J. Morris et al. (1986) to semantic objects present in the scene. The main contribution of this paper is a novel framework for integration of evidence from multiple sources with the region merging process based on the Dempster-Shafer (DS) theory by P. Smets (1988) that allows integration of sources providing evidence with different accuracy and reliability. Extensive experiments indicate that the proposed solution limits formation of regions spanning more than one semantic object.

This paper deals with coding of natural scenes in order to extract semantic information. We present a new scheme to project natural scenes onto a basis in which each dimension encodes statistically independent information. Basis extraction is performed by independent component analysis (ICA) applied to image patches culled from natural scenes. The study of the resulting coding units (coding filters) extracted from well-chosen categories of images shows that they adapt and respond selectively to discriminant features in natural scenes. Given this basis, we define global and local image signatures relying on the maximal activity of filters on the input image. Locally, the construction of the signature takes into account the spatial distribution of the maximal responses within the image. We propose a criterion to reduce the size of the space of representation for faster computation. The proposed approach is tested in the context of texture classification (111 classes), as well as natural scenes classification (11 categories, 2037 images). Using a common protocol, the other commonly used descriptors have at most 47.7% accuracy on average while our method obtains performances of up to 63.8%. We show that this advantage does not depend on the size of the signature and demonstrate the efficiency of the proposed criterion to select ICA filters and reduce the dimension.

In this paper, we evaluate the appearance tracking performance of multiple fusion schemes that combine information from standard CCTV and thermal infrared spectrum video for the tracking of surveillance objects, such as people, faces, bicycles and vehicles. We show results on numerous real world multimodal surveillance sequences, tracking challenging objects whose appearance changes rapidly. Based on these results we can determine the most promising fusion schemes

In this paper, we propose a novel method for automatic orientation of digital images. The approach is based on exploiting the properties of local statistics of natural scenes. In this way, we address some of the difficulties encountered in previous works in this area. The main contribution of this paper is to introduce a pre-classification step into carefully defined categories in order to simplify subsequent orientation detection. The proposed algorithm was tested on 9068 images and compared to existing state of the art in the area. Results show a significant improvement over previous works

This paper describes a system for object segmentation and feature extraction for surveillance video. Segmentation is performed by a dynamic vision system that fuses information from thermal infrared video with standard CCTV video in order to detect and track objects. Separate background modelling in each modality and dynamic mutual information based thresholding are used to provide initial foreground candidates for tracking. The belief in the validity of these candidates is ascertained using knowledge of foreground pixels and temporal linking of candidates. The transferable belief model is used to combine these sources of information and segment objects. Extracted objects are subsequently tracked using adaptive thermo-visual appearance models. In order to facilitate search and classification of objects in large archives, retrieval features from both modalities are extracted for tracked objects. Overall system performance is demonstrated in a simple retrieval scenario.

The efficient design of multiplierless implementations of constant matrix multipliers is challenged by the huge solution search spaces even for small scale problems. Previous approaches tend to use hill-climbing algorithms risking sub-optimal results. The proposed algorithm avoids this by exploring parallel solutions. The computational complexity is tackled by modelling the problem in a format amenable to genetic programming and hardware acceleration. Results show an improvement on state of the art algorithms with future potential for even greater savings.

This paper tackles the problem of accelerating motion estimation for video processing. A novel architecture using binary data is proposed, which attempts to reduce power consumption. The solution exploits redundant operations in the sum of absolute differences (SAD) calculation, by a mechanism known as early termination. Further data redundancies are exploited by using a run length coding addressing scheme, where access to pixels which do not contribute to the final SAD value is minimised. By using these two techniques operations and memory accesses are reduced by 93.29% and 69.17% respectively relative to a systolic array implementation.

The Fischlar-TRECVid-2004 system was developed for Dublin City University's participation in the 2004 TRECVid video information retrieval benchmarking activity. The system allows search and retrieval of video shots from over 60 hours of content. The shot retrieval engine employed is based on a combination of query text matched against spoken dialogue combined with image-image matching where a still image (sourced externally), or a keyframe (from within the video archive itself), is matched against all keyframes in the video archive. Three separate text retrieval engines are employed for closed caption text, automatic speech recognition and video OCR. Visual shot matching is primarily based on MPEG-7 low-level descriptors. The system supports relevance feedback at the shot level enabling augmentation and refinement using relevant shots located by the user. Two variants of the system were developed, one that supports both text- and image-based searching and one that supports image only search. A user evaluation experiment compared the use of the two systems. Results show that while the system combining text- and image-based searching achieves greater retrieval effectiveness, users make more varied and extensive queries with the image only based searching version.

Sensor networks are becoming increasingly important for a wide variety of applications including environmental monitoring, building safety and emergency relief services. A typical sensor network consists of a large number of small, low-power, low-cost nodes that form a self-organized network using wireless peer-to-peer communication. Because sensor networks pose unique constraints on their operation, traditional security techniques used by conventional networks cannot be applied. In this paper we consider the operational issues and security threats to sensor networks. We discuss the state of the art in terms of sensor network security and we examine the practicality of using efficient elliptic curve algorithms and identity based encryption to deploy a secure sensor network infrastructure. We evaluate the potential for realizing this on low-power, long-life devices by measuring power consumption of the operations needed for key management in a sensor network and thus provide further evidencefor the feasibility of the approach.

Technological developments in content-based analysis of digital video information are undergoing much progress, with ideas for fully automatic systems now being proposed and demonstrated. Yet because we do not yet have robust operational video retrieval systems that can be deployed and used, the usual HCI practise of conducting a usage study and an informed iterative system design is thus not possible. Físchlár-News is one of the first automatic, content-based broadcast news analysis and archival systems that process broadcast news video so that users can search, browse, and play it in an easy-to-use manner with a conventional web browser. The system incorporates a number of state-of-the-art research components, some of which are not yet considered mature technology, yet it has been built to be robust enough to be deployed to users who are interested in access to daily news throughout a university campus. In this article we report and discuss a user-evaluation study conducted with 16 users, each of whom utilized the system freely for a one month period. Results from a detailed qualitative analysis are presented, looking at collected questionnaires, incident diaries, and interaction-log data. The findings suggest that our users employed the system in conjunction with their other news update methods, such as watching TV news at home and browsing online news websites at their workplace, their major concerns being up-to-dateness and coverage of the news content. They tried to accommodate the system to fit their established web browsing habits, and they found local news content and the ability to play self-contained news stories on their desktop as major values of the system. Our study also resulted in a detailed wishlist of new features which will help in the further development of both our and others' systems.

In this paper, a hybrid technique for face detection in color images is presented. The proposed technique combines three analysis models, namely skin detection, automatic eye localization, and appearance-based face/nonface classification. Using a robust histogram-based skin detection model, skin-like pixels are first identified in the RGB color space. Based on this, face bounding-boxes are extracted from the image. On detecting a face bounding-box, approximate positions of the candidate mouth feature points are identified using the redness property of image pixels. A region-based eye localization step, based on the detected mouth feature points, is then applied to face bounding-boxes to locate possible eye feature points in the image. Based on the distance between the detected eye feature points, face/non-face classification is performed over a normalized search area using the Bayesian discriminating feature (BDF) analysis method. Some subjective evaluation results are presented on images taken using digital cameras and a Webcam, representing both indoor and outdoor scenes.

Interactive audio-visual (AV) applications such as free viewpoint video (FVV) aim to enable unrestricted spatio-temporal navigation within multiple camera environments. Current virtual viewpoint view synthesis solutions for FVV are either purely image-based implying large information redundancy; or involve reconstructing complex 3D models of the scene. In this paper we present a new multiple image view synthesis algorithm that only requires camera parameters and disparity maps. The multi-view synthesis (MVS) approach can be used in any multi-camera environment and is scalable as virtual views can be created given 1 to N of the available video inputs, providing a means to gracefully handle scenarios where camera inputs decrease or increase over time. The algorithm identifies and selects only the best quality surface areas from available reference images, thereby reducing perceptual errors in virtual view reconstruction. Experimental results are presented and verified using both objective (PSNR) and subjective comparisons.

In this paper, we propose a generic framework for event detection in broadcast video of multiple different field-sports. Features indicating significant events are selected, and robust detectors built. These features are rooted in generic characteristics common to all genres of field-sports. The evidence gathered by the feature detectors is combined by means of a support vector machine, which infers the occurrence of an event based on a model generated during a training phase. The system is tested across multiple genres of field-sports including soccer, rugby, hockey and Gaelic football and the results suggest that high event retrieval and content rejection statistics are achievable.

One of the main aims of emerging audio-visual (AV) applications is to provide interactive navigation within a captured event or scene. This paper presents a view synthesis algorithm that provides a scalable and flexible approach to virtual viewpoint synthesis in multiple camera environments. The multi-view synthesis (MVS) process consists of four different phases that are described in detail: surface identification, surface selection, surface boundary blending and surface reconstruction. MVS view synthesis identifies and selects only the best quality surface areas from the set of available reference images, thereby reducing perceptual errors in virtual view reconstruction. The approach is camera setup independent and scalable as virtual views can be created given 1 to N of the available video inputs. Thus, MVS provides interactive AV applications with a means to handle scenarios where camera inputs increase or decrease over time.

In this paper, we propose a novel audio-visual feature-based framework for event detection in broadcast video of multiple different field sports. Features indicating significant events are selected and robust detectors built. These features are rooted in characteristics common to all genres of field sports. The evidence gathered by the feature detectors is combined by means of a support vector machine, which infers the occurrence of an event based on a model generated during a training phase. The system is tested generically across multiple genres of field sports including soccer, rugby, hockey, and Gaelic football and the results suggest that high event retrieval and content rejection statistics are achievable.

The paper presents an approach to segmenting broadcast TV news programmes automatically into individual news stories. We first segment the programme into individual shots, and then a number of analysis tools are run on the programme to extract features to represent each shot. The results of these feature extraction tools are then combined using a support vector machine trained to detect anchorperson shots. A news broadcast can then be segmented into individual stories based on the location of the anchorperson shots within the programme. We use one generic system to segment programmes from two different broadcasters, illustrating the robustness of our feature extraction process to the production styles of different broadcasters.

In this paper, we discuss the criteria that should be satisfied by a descriptor for nonrigid shapes with a single closed contour. We then propose a shape representation method that fulfills these criteria. In the proposed approach, contour convexities and concavities at different scale levels are represented using a two-dimensional (2-D) matrix. The representation can be visualized as a 2-D surface, where "hills" and "valleys" represent contour convexities and concavities, respectively. The optimal matching of two shape representations is achieved using dynamic programming and a dissimilarity measure is defined based on this matching. The proposed algorithm is very efficient and invariant to several kinds of transformations including some articulations and modest occlusions. The retrieval performance of the approach is illustrated using the MPEG-7 shape database, which is one of the most complete shape databases currently available. Our experiments indicate that the proposed representation is well suited for object indexing and retrieval in large databases. Furthermore, the representation can be used as a starting point to obtain more compact descriptors.

The extensive amount of media coverage today, generates difficulties in identifying and selecting desired information. Browsing and retrieval systems become more and more necessary in order to support users with powerful and easy-to-use tools for searching, browsing and summarization of information content. The starting point for these tasks in video browsing and retrieval systems is the low level analysis of video content, especially the segmentation of video content into shots. This paper presents a fast and efficient way to detect shot changes using only the temporal distribution of macroblock types in MPEG compressed video. The notion of a dominant reference frame is introduced here. A dominant frame denotes the reference frame (I or P) used as prediction reference for most of the macroblocks from a subsequent B frame.

The frequency subband scale-factors are fundamental components of MPEG-1 audio encoded bitstreams. Examination of scale-factor weights is sufficient for the establishment of an audio amplitude profile of an audio track. If, for sports programme TV broadcasts, the audio amplitude is assumed to primarily reflect the noise level exhibited by the commentator (and/or attending spectators), then, this vocal reaction to the significance of unfolding events may be used as a basis for summarisation. i.e. by relying on the exhilaration, or otherwise, expressed by the commentator/spectators, individual clips of the programme (e.g. camera shots), may be ranked according to their relative significance. A summary may then be produced by amalgamating (chronologically) any number of these clips corresponding to selected audio peaks.

This paper describes a demonstration system which automatically indexes broadcast television content for subsequent non-linear browsing. User-specified television programmes are captured in MPEG-1 format and analysed using a number of video indexing tools such as shot boundary detection, keyframe extraction, shot clustering and news story segmentation. A number of different interfaces have been developed which allow a user to browse the visual index created by these analysis tools. These interfaces are designed to facilitate users locating video content of particular interest. Once such content is located, the MPEG-1 bitstream can be streamed to the user in real-time. This paper describes both the high-level functionality of the system and the low-level indexing tools employed, as well as giving an overview of the different browsing mechanisms employed.

This paper presents an approach to segmenting individual news stories in broadcast news programmes. The approach first performs shot boundary detection and keyframe extraction on the programme. Shots are then clustered into groups based on their colour and temporal similarity. The clustering process is controlled using the groups' statistics. After clustering, a set of criteria are applied and groups are successively eliminated in order to converge upon a set of anchorperson groups. The temporal locations of the shots in these anchorperson groups are then used to segment the programme in terms of individual news items. This work is carried out within the context of a complete video indexing, browsing and retrieval system.

This paper presents an object-based segmentation and tracking scheme for video sequences. The probability density function (PDF) of each image to be segmented is modeled as a mixture of independent object PDF. In the first image of the sequence, the parameters of the mixture are initially estimated based on user interaction. These parameters are then iteratively updated using the expectation maximization (EM) algorithm. A classification procedure applied to the results of the EM algorithm allows an object-based segmentation of the image to be constructed. In subsequent images of the sequence, the segmented objects are automatically tracked using motion estimation/compensation and a similar EM-based segmentation framework. Results indicate that accurate object segmentations and robust tracking can be obtained using this flexible approach.

This paper proposes two description schemes (DSs) to describe the visual information of an audio-visual (AV) document. The first one, is devoted to still images. It describes the image visual appearance and its structure with regions as well as its semantic content in terms of objects. The second DS is devoted to video sequences. It describes the sequence structure as well as its semantic content in terms of events. Features such as motion, camera activity, etc. are included in this DS. Moreover, it involves static visual representations such as key-frames, background mosaics and key-regions. These elements are considered as still images and are described by the first DS.

This paper addresses the segmentation and subsequent tracking of the moving objects within a video sequence. The approach is to jointly produce segmentations and motion parameters which minimise the interframe coding rate of the video sequence. This minimisation is performed using a previously proposed framework based on the expectation-maximisation (EM) algorithm and the minimum description length (MDL) estimate. The paper concentrates on extensions of this framework which have been incorporated to ensure spatial and temporal coherence in the output segmentation sequence. The work has been conducted considering video coding applications and in particular, the MPEG-4 standardisation effort.