2D cameras are often used in interactive systems. Other systems like gaming consoles provide more powerful 3D cameras for short range depth sensing. Overall, these cameras are not reliable in large, complex environments. In this work, we propose a 3D stereo vision based pipeline for interactive systems, that is able to handle both ordinary and sensitive applications, through robust scene understanding. We explore the fusion of multiple 3D cameras to do full scene reconstruction, which allows for preforming a wide range of tasks, like event recognition, subject tracking, and notification. Using possible feedback approaches, the system can receive data from the subjects present in the environment, to learn to make better decisions, or to adapt to completely new environments. Throughout the paper, we introduce the pipeline and explain our preliminary experimentation and results. Finally, we draw the roadmap for the next steps that need to be taken, in order to get this pipeline into production.

This study explores the potential of enhancing interaction experiences, such as virtual reality (VR) games, through the use of computational attention models. Our proposed approach utilizes a saliency map generated by attention models to dynamically adjust game difficulty levels and to help in the game level design, resulting in a more immersive and engaging experience for users. To inform the development of this approach, we present an experimental setup that is able tp collect data in a VR environment and intends to be able to validate the adaptation of attention models to this domain. Through this work, we aim to create a framework for VR game design that leverages attention models to offer a new level of immersion and engagement for users. We believe our contributions have significant potential to enhance VR experiences and advance the field of game design.

Human audience analysis is crucial in numerous domains to understand people’s behavior based on their knowledge and environment. In this paper we focus on simulation-based training which has become a popular teaching approach that requires a trainer able to manage a lot of data at the same time. We present tools that are currently being developed to help trainers from two different fields: training for practical teaching gestures and training in civil defense. In this sense, three technological blocks are built to collect and analyze data about trainers’ and learners’ gestures, gaze, speech and movements. The paper also discusses the future work planned for this project, including the integration of the framework into the Noldus system and its use in civil security training. Overall, the article highlights the potential of technology to improve simulation-based training and provides a roadmap for future development.

We present ActioNet, a groundbreaking lightweight neural network architecture optimized for action recognition tasks, particularly in resource-constrained environments such as drones and edge devices. Utilizing a strategically modified 3D U-Net encoder followed by fully connected layers for fine-grained classification, ActioNet manages to achieve a promising validation accuracy of 72%. This is accomplished with a notably compact model size of just 46MB, making it uniquely suitable for devices with limited computational capabilities. Although ActioNet may not surpass state-of-the-art models in terms of sheer accuracy, it distinguishes itself through its fast inference times and small footprint. These attributes make real-time action recognition not only feasible but also efficient in constrained operational settings. We argue that ActioNet serves as a meaningful contribution to the emerging field of efficient deep learning and provides a solid foundation for future advancements in lightweight action recognition models.

In this paper, we deal with the problem of shadow detection of UAVs, which impacts their navigation.We propose to generate synthetic images containing shadows in random locations, backgrounds, sizes, and opacities in order to augment our dataset.The generated data is used to train and compare several models to effectively detect, in real-time, UAVs shadows which will help to stabilize their localization and navigation.Deep learning models such as SSD, YOLOv3, and YOLOv5 are tested for the detection part.With our approach, we achieved 99% of the mean average precision when using the YOLOv5.

Introducing sparsity in a convnet has been an efficient way to reduce its complexity while keeping its performance almost intact. Most of the time, sparsity is introduced using a three-stage pipeline: 1) training the model to convergence, 2) pruning the model, 3) fine-tuning the pruned model to recover performance. The last two steps are often performed iteratively, leading to reasonable results but also to a time-consuming process. In our work, we propose to remove the first step of the pipeline and to combine the two others in a single training-pruning cycle, allowing the model to jointly learn the optimal weights while being pruned. We do this by introducing a novel pruning schedule, named One-Cycle Pruning (OCP), which starts pruning from the beginning of the training, and until its very end. Experiments conducted on a variety of combinations between architectures (VGG-16, ResNet-18), datasets (CIFAR-10, CIFAR-100, Caltech-101), and sparsity values (80%, 90%, 95%) show that not only OCP consistently outperforms common pruning schedules such as One-Shot, Iterative and Automated Gradual Pruning, but also that it drastically reduces the required training budget. More-over, experiments following the Lottery Ticket Hypothesis show that OCP allows to find higher quality and more stable pruned networks.

Human eye movements are far from being well described with current indicators. From the dataset provided by the ETRA 2019 challenge, we analyzed saccades and fixations during a free exploration of blank or natural scenes and during visual search. Based on the two modes of exploration, ambient and focal, we used the K coefficient [Krejtz et al. 2016]. We failed to find any differences between tasks but this indicator gives only the dominant mode over the entire recording. The stability of both modes, assesses with the switch frequency and the mode duration allowed to differentiate gaze behavior according to situations. Time course analyses of K coefficient and switch frequency corroborate that the latter is a useful indicator, describing a greater portion of the eye movement recording.

Visual attention allows the human visual system to effectively deal with the huge flow of visual information acquired by the retina. Since the years 2000, the human visual system began to be modelled in computer vision to predict abnormal, rare and surprising data. Attention is a product of the continuous interaction between bottom-up (mainly feature-based) and top-down (mainly learning-based) information. Deep-learning (DNN) is now well established in visual attention modelling with very effective models. The goal of this paper is to investigate the importance of bottom-up versus top-down attention. First, we enrich with top-down information classical bottom-up models of attention. Then, the results are compared with DNN-based models. Our provocative question is: “do deep-learning saliency models really predict saliency or they simply detect interesting objects?”. We found that if DNN saliency models very accurately detect top-down features, they neglect a lot of bottom-up information which is surprising and rare, thus by definition difficult to learn.

Although many visual attention models have been proposed, very few saliency models investigated the impact of audio information. To develop audio-visual attention models, researchers need to have a ground truth of eye movements recorded while exploring complex natural scenes in different audio conditions. They also need tools to compare eye movements and gaze patterns between these different audio conditions. This paper describes a toolbox that answer these needs by proposing a new eye-tracking dataset and its associated analysis ToolBox that contains common metrics to analysis eye movements. Our eye-tracking dataset contains the eye positions gathered during four eye-tracking experiments. A total of 176 observers were recorded while exploring 148 videos (mean duration = 22 s) split between different audio conditions (with or without sound) and visual categories (moving objects, landscapes and faces). Our ToolBox allows to visualize the temporal evolution of different metrics computed from the recorded eye positions. Both dataset and ToolBox are freely available to help design and assess visual saliency models for audiovisual dynamic stimuli.

Saliency models provide heatmaps highlighting the probability of each pixel to attract human gaze. To define image's important regions, features maps are extracted. The rarity, surprise or contrast are computed leading to conspicuity maps, showing important regions of each feature map. The final saliency map is obtained by merging these maps. The fusion process is usually a linear combination of the maps where the coefficients show their importance. We propose a novel generic fusion mechanism based on 1) using a rarity-based attention module and 2) using neural networks to achieve the fusion. The first layer of the NN merges the weighted feature maps into a saliency map. The second layer takes into account the spatial information. The approach is compared to 8 models using 4 different comparison metrics on open state-of-the-art databases.

Modeling human attention has been arousing a lot of interest due to its numerous applications. The process that allows us to focus on some more important stimuli is defined as the “attention”. Seam carving is an approach to resize images or video sequences while preserving the semantic content. To define what is important, gradient was first used but due to limitations of this approach, saliency models, which are able to predict whether regions in the images attract human attention, are now used. Most of them are optimized to conform a ground truth like eye tracking but there is no way to know the efficiency of the saliency models applied in a specific application like in seam carving. In this paper, we propose a criterion, based on the quantity of geometric deformation and the image's reduction, which evaluate the quality of a resizing by seam carving. This criterion is applied on evaluation of image (SCES) or video (SCED) resizing. We validate our criterion with subjective evaluation and used it to rank state of the art saliency models for seam carving. Evaluation of the image by SCES gives a Spearman correlation of −0.92196 and a Pearson correlation of −0.8812. For the video, the final SCED gives a Spearman correlation of −0.81351 and a Pearson correlation of −0.80581.

Visual saliency has been an increasingly active research area in the last ten years with dozens of saliency models recently published. Nowadays, one of the big challenges in the field is to find a way to fairly evaluate all of these models. In this paper, on human eye fixations, we compare the ranking of 12 state-of-the art saliency models using 12 similarity metrics. The comparison is done on Jian Li's database containing several hundreds of natural images. Based on Kendall concordance coefficient, it is shown that some of the metrics are strongly correlated leading to a redundancy in the performance metrics reported in the available benchmarks. On the other hand, other metrics provide a more diverse picture of models' overall performance. As a recommendation, three similarity metrics should be used to obtain a complete point of view of saliency model performance.

In this paper, a new spatio-temporal saliency model is presented. Based on the idea that both spatial and temporal features are needed to determine the saliency of a video, this model builds upon the fact that locally contrasted and globally rare features are salient. The features used in the model are both spatial (color and orientations) and temporal (motion amplitude and direction) at several scales. To be more robust to moving camera a module computes the global motion and to be more consistent in time, the saliency maps are combined together after a temporal filtering. The model is evaluated on a dataset of 24 videos split into 5 categories (Abnormal, Surveillance, Crowds, Moving camera, and Noisy). This model achieves better performance when compared to several state-of-the-art saliency models.

The image memorability consists in the faculty of an image to be recalled after a period of time. Recently, the memorability of an image database was measured and some factors responsible for this memorability were highlighted. In this paper, we investigate the role of visual attention in image memorability around two axis. The first one is experimental and uses results of eye-tracking performed on a set of images of different memorability scores. The second investigation axis is predictive and we show that attention-related features can advantageously replace low-level features in image memorability prediction. From our work it appears that the role of visual attention is important and should be more taken into account along with other low-level features.

In this paper, four new approaches of global context recognition algorithms (gist) are introduced. They are able to automatically distinguish context differences like buildings, coast, home (indoor), mountain or streets. All proposed models are biologically plausible and are able to deal with both color and gray-level images. They use Gabor or Log-Gabor filters to extract features that better mimic human visual perception. Those features are then classified using a Mahalanobis space (when a subset of features is extracted) or in a high-dimensional Gaussian space (when all features are taken into account) with Support Vector Machines (SVM). The proposed models are compared to a standard state of the art gist model to proof their efficiency.

In this paper, a new bottom-up visual saliency model is proposed. Based on the idea that locally contrasted and globally rare features are salient, this model will be called “RARE” in the following sections. It uses a sequential bottom-up features extraction where first low-level features as luminance and chrominance are computed and from those results medium-level features as image orientations are extracted. A qualitative and a quantitative comparison are achieved on a 120 images dataset. The RARE algorithm powerfully predicts human fixations compared with most of the freely available saliency models.

This paper deals with the selection of relevant motion from multi-object movement. The proposed method is based on a multi-scale approach using features extracted from optical flow and global rarity quantification to compute bottom-up saliency maps. It shows good results from four objects to dense crowds with increasing performance. The results are convincing on synthetic videos, simple real video movements, a pedestrian database and they seem promising on very complex videos with dense crowds. This algorithm only uses motion features (direction and speed) but can be easily generalized to other dynamic or static features. Video surveillance, social signal processing and, in general, higher level scene understanding can benefit from this method.

This paper describes a simple and "pre-cortical" visual attention model, which does not take image directions into account. We compute rarity-based saliency maps and then we describe the relation between texture and visual attention. Finally we decompose the image into several textures with different regularities. Our purpose is to compress textures into images using small repeating patterns.

Our research deals with a fully automatic and fast localization of possible tumoral areas on computed tomography scanner (CT scan) images. The aim of this method is not to segment tumors but only to highlight the areas where tumor has the greater probability to be located. To achieve this task, we use the bilateral symmetry of the human body and the asymmetry introduced by the presence of tumors. Our work was initially dedicated to the head and neck area but it should work well for any other body part and even better for more symmetric areas like the brain.

In this article, we present a novel fully automatic character segmentation for camera-based images. This is a top-down approach inspired by the human visual system: the high level step is based on color clustering while the low level one on the phase of log-Gabor filters. First, this latter step deals with remaining touching characters after the color clustering approach and then takes into account the neighborhood information to address the problem of broken characters. Results are very encouraging and the method should be fast enough to be used on embedded platforms like personal digital assistants.

Our research deals with a fully automatic and fast visualization of possible tumoral areas on CT Scan images. To achieve this task, we use the bilateral symmetry of the human body and the asymmetry introduced by the presence of tumors. Our work was initially dedicated to the head and neck area but it should work well for any other body part and even better for more symmetric areas like the brain.