The current paradigm of training deep neural networks relies on large, annotated and representative datasets. They assume a static world where the target domain does not change. However, in the real-world, data changes over time and is often available on the fly. Naive retraining on new data causes catastrophic forgetting and the network is unable to generalize on old data. Streaming learning is a type of incremental learning where networks learn sequentially and as soon as a sample is available from the data stream. Instead of training on every new sample, we propose an uncertainty based selection criteria to improve our previously proposed fast streaming learning method Move-to-Data (MTD), called Entropy-based MTD (EMTD). Besides, streaming learning methods have so far mostly used Convolutional Neural Networks (CNNs) but in recent times Vision Transformers (ViTs) have shown much better performances for many vision tasks. Therefore, we use ViT based Video Transformer to analyse MTD, EMTD and their gradient descent based "retargeting" steps. We have compared the performances of EMTD with MTD (w/wo retargeting) and a popular streaming learning method ExStream for the transformer. EMTD is able to outperform baseline MTD, and EMTD with retargeting achieves close results as ExStream and is ∼ 1.2 times faster.

The paper proposes a video transformer architecture for detection of risk events on frail adults with ego video monitoring data. First we introduce an extended taxonomy for risk events, and then we propose a transformer based video recognition model for detection of these risk events. The proposed transformer architecture consists of separable attention for spatial and temporal data. We also introduce a pooling operation on the temporal video data by learning of their importance. The experiments have been conducted on visual data of in-the-wild recorded BIRDS dataset and on Kinetics-400 for benchmarking. The use of the pooling operation in transformers gives an increment of 3% on BIRDS dataset.

Multimedia today is also in multimodality. Working with heterogeneous signals we use multimedia techniques of data fusion and mining. Classification from real world datasets are often challenging. The paper is devoted to the detection of personal risk situations of fragile people from multi-modal sensing real world lifelog data named BIRDS. Using a real-world data is challenging as the risk situations are rare and last just a few seconds compared to the global volume of the dataset. In this paper we propose a GRU architecture with global attention block to recognise semantic risk situations from a limited taxonomy. Attention is also focused on data organisation and pre-processing with imputation and normalisation. The proposed method is applied to a real-world collected multimodal dataset and to the OpenSource dataset UCI-HAR for the sake of comparison with the state-of-the-art.

The automatic recognition of risk situations for frail people is an urgent research topic for the interdisciplinary artificial intelligence and multimedia community. Risky situations can be recognized from lifelog data recorded with wearable devices. In this paper, we present a new approach for the detection of semantic risk situations for frail people in lifelog data. Concept matching between general lifelog and risk taxonomies was realized and tuned AlexNet was deployed for detection of two semantic risks situations such as risk of domestic accident and risk of fraud with promising results.

The ageing of the world population has raised ever-increasing demands for measuring physical conditions and assisting the elderly and/or fragile population at their homes. Physiological, motor and even environmental measurements are excellent indicators of their health status. Furthermore, connected wearable technologies in the framework of the Internet of Things allow for risk situations prevention. Recent advances of Artificial Intelligence techniques make possible high-accuracy decision making on multi-sensory data. Nevertheless, to train models and to perform online real-time detection of risk events from heterogeneous taxonomy, robust wearable devices are required. In this paper, existing solutions are reviewed for human sensing for these purposes. Moreover, we present the implementation of a multi-sensor device for the recognition of risk situations with a focus on the data synchronisation.