The computer vision community has explored dyadic interactions for atomic actions such as pushing, carrying-object, etc. However, with the advancement in deep learning models, there is a need to explore more complex dyadic situations such as loose interactions. These are interactions where two people perform certain atomic activities to complete a global action irrespective of temporal synchronisation and physical engagement, like cooking-together for example. Analysing these types of dyadic-interactions has several useful applications in the medical domain for social-skills development and mental health diagnosis. To achieve this, we propose a novel dual-path architecture to capture the loose interaction between two individuals. Our model learns global abstract features from each stream via a CNNs backbone and fuses them using a new Global-Layer-Attention module based on a cross-attention strategy. We evaluate our model on real-world autism diagnoses such as our Loose-Interaction dataset, and the publicly available Autism dataset for loose interactions. Our network achieves baseline results on the Loose-Interaction and SOTA results on the Autism datasets. Moreover, we study different social interactions by experimenting on a publicly available dataset i.e. NTU-RGB+D (interactive classes from both NTU-60 and NTU-120). We have found that different interactions require different network designs. We also compare a slightly different version of our method (details in Section 3.6) by incorporating time information to address tight interactions achieving SOTA results.

Seizure events feature temporary abnormalities in muscle control or movements. They are usually caused by excessive neuronal activities in the brain, and are called epileptic seizures (ES). Nevertheless, not all seizures are epileptic in origin. Some are caused by psychological reasons, and such type of seizures are called psychogenic non-epileptic seizures (PNES). We propose a method to classify ES and PNES based on clinical signs in the seizure videos. In particular, inspired by BERT, we propose a Transformer-based framework that pre-trains on large unlabeled clinical videos, and then we fine-tune the pre-trained model for seizure classification with a minimum modification. We conduct a leave-one-subject-out (LOSO) validation on our dataset. The F1-score and accuracy are 0.82 and 0.75, respectively. To our knowledge, the proposed approach is the first attempt to use large unannotated data and learn useful representations for downstream tasks in the field of video based seizure analysis.

In this work, we propose a multi-stream approach with knowledge distillation to classify epileptic seizures and psychogenic non-epileptic seizures. The proposed framework utilizes multi-stream information from keypoints and appearance from both body and face. We take the detected keypoints through time as spatio-temporal graph and train it with an adaptive graph convolutional networks to model the spatio-temporal dynamics throughout the seizure event. Besides, we regularize the keypoint features with complementary information from the appearance stream by imposing a knowledge distillation mechanism. We demonstrate the effectiveness of our approach by conducting experiments on real-world seizure videos. The experiments are conducted by both seizure-wise cross validation and leave-one-subject-out validation, and with the proposed model, the performances of the F1-scorelaccuracy are 0.89/0.87 for seizure-wise cross validation, and 0.75/0.72 for leave-one-subject-out validation.

In this paper, we introduce a new approach for Activities of Daily Living (ADL) recognition. In order to discriminate between activities with similar appearance and motion, we focus on their temporal structure. Actions with subtle and similar motion are hard to disambiguate since long-range temporal information is hard to encode. So, we propose an end-to-end Temporal Model to incorporate long-range temporal information without losing subtle details. The temporal structure is represented globally by different temporal granularities and locally by temporal segments. We also propose a two-level pose driven attention mechanism to take into account the relative importance of the segments and granularities. We validate our approach on 2 public datasets: a 3D human activity dataset (NTU-RGB+D) and a human-object interaction dataset (Northwestern-UCLA Multiview Action 3D). Our Temporal Model can also be incorporated with any existing 3D CNN (including attention based) as a backbone which reveals its robustness.

In this paper, we present a new attention model for the recognition of human action from RGB-D videos. We propose an attention mechanism based on 3D articulated pose. The objective is to focus on the most relevant body parts involved in the action. For action classification, we propose a classification network compounded of spatio-temporal subnetworks modeling the appearance of human body parts and RNN attention subnetwork implementing our attention mechanism. Furthermore, we train our proposed network end-to-end using a regularized cross-entropy loss, leading to a joint training of the RNN delivering attention globally to the whole set of spatio-temporal features, extracted from 3D ConvNets. Our method outperforms the State-of-the-art methods on the largest human activity recognition dataset available to-date (NTU RGB+D Dataset) which is also multi-views and on a human action recognition dataset with object interaction (Northwestern-UCLA Multiview Action 3D Dataset).

In this study, the authors propose a complete framework based on a hierarchical activity model to understand and recognise activities of daily living in unstructured scenes. At each particular time of a long‐time video, the framework extracts a set of space‐time trajectory features describing the global position of an observed person and the motion of his/her body parts. Human motion information is gathered in a new feature that the authors call perceptual feature chunks (PFCs). The set of PFCs is used to learn, in an unsupervised way, particular regions of the scene (topology) where the important activities occur. Using topologies and PFCs, the video is broken into a set of small events (‘primitive events’) that have a semantic meaning. The sequences of ‘primitive events’ and topologies are used to construct hierarchical models for activities. The proposed approach has been tested with the medical field application to monitor patients suffering from Alzheimer's and dementia. The authors have compared their approach to their previous study and a rule‐based approach. Experimental results show that the framework achieves better performance than existing works and has the potential to be used as a monitoring tool in medical field applications.

The tracking algorithm performance depends on video content. This paper presents a new multi-object tracking approach which is able to cope with video content variations. First the object detection is improved using Kanade-Lucas-Tomasi (KLT) feature tracking. Second, for each mobile object, an appropriate tracker is selected among a KLT-based tracker and a discriminative appearance-based tracker. This selection is supported by an online tracking evaluation. The approach has been experimented on three public video datasets. The experimental results show a better performance of the proposed approach compared to recent state of the art trackers.

We propose a new representation of videos, as spatio-temporal fibers. These fibers are clusters of trajectories that are meshed spatially in the image domain. They form a hierarchical partition of the video into regions that are coherent in time and space. They can be seen as dense, spatially-organized, long-term optical flow. Their robustness to noise and ambiguities is ensured by taking into account the reliability of each source of information. As fibers allow users to handle easily moving objects in videos, they prove useful for video editing, as demonstrated in a video inpainting example.

Population aging has been motivating academic research and industry to develop technologies for the improvement of older people's quality of life, medical diagnosis, and support on frailty cases. Most of available research prototypes for older people monitoring focus on fall detection or gait analysis and rely on wearable, environmental, or video sensors. We present an evaluation of a research prototype of a video monitoring system for event recognition of older people. The prototype accuracy is evaluated for the recognition of physical tasks (e.g., Up and Go test) and instrumental activities of daily living (e.g., watching TV, writing a check) of participants of a clinical protocol for Alzheimer's disease study (29 participants). The prototype uses as input a 2D RGB camera, and its performance is compared to the use of a RGB-D camera. The experimentation results show the proposed approach has a competitive performance to the use of a RGB-D camera, even outperforming it on event recognition precision. The use of a 2D-camera is advantageous, as the camera field of view can be much larger and cover an entire room where at least a couple of RGB-D cameras would be necessary.

Activity recognition has been a growing research topic in the last years and its application varies from automatic recognition of social interaction such as shaking hands, parking lot surveillance, traffic monitoring and the detection of abandoned luggage. This paper describes a probabilistic framework for uncertainty handling in a description-based event recognition approach. The proposed approach allows the flexible modeling of composite events with complex temporal constraints. It uses probability theory to provide a consistent framework for dealing with uncertain knowledge for the recognition of complex events. We validate the event recognition accuracy of the proposed algorithm on real-world videos. The experimental results show that our system can successfully recognize activities with a high recognition rate. We conclude by comparing our algorithm with the state of the art and showing how the definition of event models and the probabilistic reasoning can influence the results of real-time event recognition.

The present work introduces a new method for activity extraction from video. To achieve this, we focus on the modelling of context by developing an algorithm that automatically learns the main activity zones of the observed scene by taking as input the trajectories of detected mobiles. Automatically learning the context of the scene (activity zones) allows first to extract a knowledge on the occupancy of the different areas of the scene. In a second step, learned zones are employed to extract people activities by relating mobile trajectories to the learned zones, in this way, the activity of a person can be summarised as the series of zones that the person has visited. For the analysis of the trajectory, a multiresolution analysis is set such that a trajectory is segmented into a series of tracklets based on changing speed points thus allowing differentiating when people stop to interact with elements of the scene or other persons. Tracklets allow thus to extract behavioural information. Starting and ending tracklet points are fed to a simple yet advantageous incremental clustering algorithm to create an initial partition of the scene. Similarity relations between resulting clusters are modeled employing fuzzy relations. These can then be aggregated with typical soft-computing algebra. A clustering algorithm based on the transitive closure calculation of the fuzzy relations allows building the final structure of the scene. To allow for incremental learning and update of activity zones (and thus people activities), fuzzy relations are defined with online learning terms. We present results obtained on real videos from different activity domains.

This paper addresses the problem of multi-target tracking in crowded scenes from a single camera. We propose an algorithm for learning discriminative appearance models for different targets. These appearance models are based on covariance descriptor extracted from tracklets given by a short-term tracking algorithm. Short-term tracking relies on object descriptors tuned by a controller which copes with context variation over time. We link tracklets by using discriminative analysis on a Riemannian manifold. Our evaluation shows that by applying this discriminative analysis, we can reduce false alarms and identity switches, not only for tracking in a single camera but also for matching object appearances between non-overlapping cameras.

Human re-identification is defined as a requirement to determine whether a given individual has already appeared over a network of cameras. This problem is particularly hard by significant appearance changes across different camera views. In order to re-identify people a human signature should handle difference in illumination, pose and camera parameters. We propose a new appearance model combining information from multiple images to obtain highly discriminative human signature, called Mean Riemannian Covariance Grid (MRCG). The method is evaluated and compared with the state of the art using benchmark video sequences from the ETHZ and the i-LIDS datasets. We demonstrate that the proposed approach outperforms state of the art methods. Finally, the results of our approach are shown on two other more pertinent datasets.

In many surveillance systems there is a requirement to determine whether a given person of interest has already been observed over a network of cameras. This paper presents two approaches for this person re-identification problem. In general the human appearance obtained in one camera is usually different from the ones obtained in another camera. In order to re-identify people the human signature should handle difference in illumination, pose and camera parameters. Our appearance models are based on hoar-like features and dominant color descriptors. The AdaBoost scheme is applied to both descriptors to achieve the most invariant and discriminative signature. The methods are evaluated using benchmark video sequences with different camera views where people are automatically detected using Histograms of Oriented Gradients (HOG). The reidentification performance is presented using the cumulative matching characteristic (CMC) curve.

In many surveillance systems there is a requirement to determine whether a given person of interest has already been observed over a network of cameras. This is the person re-identification problem. The human appearance obtained in one camera is usually different from the ones obtained in another camera. In order to re-identify people the human signature should handle difference in illumination, pose and camera parameters. We propose a new appearance model based on spatial covariance regions extracted from human body parts. The new spatial pyramid scheme is applied to capture the correlation between human body parts in order to obtain a discriminative human signature. The human body parts are automatically detected using Histograms of Oriented Gradients (HOG). The method is evaluated using benchmark video sequences from i-LIDS Multiple-Camera Tracking Scenario data set. The re-identification performance is presented using the cumulative matching characteristic (CMC) curve. Finally, we show that the proposed approach outperforms state of the art methods.

HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et a ̀ la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

This paper presents a constraint-based approach for video event recognition with probabilistic reasoning for handling uncertainty. The main advantage of constraint-based approaches is the possibility for human expert to model composite events with complex temporal constraints. But the approaches are usually deterministic and do not enable the convenient mechanism of probability reasoning to handle the uncertainty. The first advantage of the proposed approach is the ability to model and recognize composite events with complex temporal constraints. The second advantage is that probability theory provides a consistent framework for dealing with uncertain knowledge for a robust and reliable recognition of complex event. This approach is evaluated with 4 real healthcare videos and a public video ETISEO'06. The results are compared with state of the art method. The comparison shows that the proposed approach improves significantly the process of recognition and characterizes the likelihood of the recognized events.

Since the population of the elderly grows highly, the improvement of the quality of life of elderly at home is of a great importance. This can be achieved through the development of technologies for monitoring their activities at home. In this context, we propose an activity monitoring system which aims to achieve behavior analysis of elderly people. The proposed system consists of an approach combining heterogeneous sensor data to recognize activities at home. This approach combines data provided by video cameras with data provided by environmental sensors attached to house furnishings. In this paper, we validate the proposed activity monitoring system for the recognition of a set of daily activities (e.g. using kitchen equipment, preparing meal) for 9 real elderly volunteers living in an experimental apartment. We compare the behavioral profile between the 9 elderly volunteers. This study shows that the proposed system is thoroughly accepted by the elderly and it is also well appreciated by the medical staff.

This article is concerned with on-line counting of harmful insects of certain species in videos in the framework of in situ video-surveillance that aims at the early detection of prominent pest attacks in greenhouse crops. The video-processing challenges that need to be coped with concern mainly the low spatial resolution and color contrast of the objects of interest in the videos, the outdoor issues and the video-processing which needs to be done in quasi-real time. Thus, we propose an approach which makes use of a pattern recognition algorithm to extract the locations of the harmful insects of interest in a video, which we combine with some video-processing algorithms in order to achieve an on-line video-surveillance solution. The system has been validated off-line on the whiteflie species (one potential harmful insect) and has shown acceptable performance in terms of accuracy versus computational time.

The present work presents a novel approach for activity extraction and knowledge discovery from video. Spatial and temporal properties from detected mobile objects are modeled employing fuzzy relations. These can then be aggregated employing typical soft-computing algebra. A clustering algorithm based on the transitive closure calculation of the fuzzy relations allows finding spatio-temporal patterns of activity. We employ trajectory-based analysis of mobiles in the video to discover the points of entry and exit of mobiles appearing in the scene and ultimately deduce the different areas of activity in the scene. These areas can be reported as activity maps with different granularities thanks to the analysis of the transitive closure matrix of the mobile fuzzy spatial relations. Discovered activity zones and spatio-temporal patterns of activity can be labeled in a human-like language. We present results obtained on real videos corresponding to apron monitoring in the Toulouse airport in France.

In this paper we propose a new multisensor based activity recognition approach which uses video cameras and environmental sensors in order to recognize interesting elderly activities at home. This approach aims to provide accuracy and robustness to the activity recognition system. In the proposed approach, we choose to perform fusion at the high-level (event level) by combining video events with environmental events. To measure the accuracy of the proposed approach, we have tested a set of human activities in an experimental laboratory. The experiment consists of a scenario of daily activities performed by fourteen volunteers (aged from 60 to 85 years). Each volunteer has been observed during 4 hours and 14 video scenes have been acquired by 4 video cameras (about ten frames per second). The fourteen volunteers were asked to perform a set of household activities, such as preparing a meal, taking a meal, washing dishes, cleaning the kitchen, and watching TV. Each volunteer was alone in the laboratory during the experiment.

In this paper, we propose an approach for surveillance video indexing and retrieval. The objective of this approach is to answer five main challenges we have met in this domain: (1) the lack of means for finding data from the indexed databases, (2) the lack of approaches working at different abstraction levels, (3) imprecise indexing, (4) incomplete indexing, (5) the lack of user-centered search. We propose a new data model containing two main types of extracted video contents: physical objects and events. Based on this data model, we present a new rich and flexible query language. This language works at different abstraction levels, provides both exact and approximate matching and takes into account users' interest. In order to work with the imprecise indexing, two new methods respectively for object representation and object matching are proposed. Videos from two projects which have been partially indexed are used to validate the proposed approach. We have analyzed both query language usage and retrieval results. The obtained retrieval results analyzed by the average normalized ranks are promising. The retrieval results at the object level are compared with another state of the art approach.

In this paper, we propose a shadow removal algorithm for indoor scenes. This algorithm uses three types of constraints: chromaticity consistency, texture consistency and range of shadow intensity. The chromaticity consistency is verified in both HSV and RGB color spaces. The texture verification is based on the local coherency (over a pixel neighbourhood) of intensity reduction ratio between shadows and background. Finally, for the range of shadow intensity, we define a localized lower bound of the intensity reduction ratio so that dark mobile objects are not classified as shadows. Because the chromaticity constraint is only correct if the chromaticity of ambient light is the same as that of diffuse light, our algorithms only works in the indoor scenes.

Extracting automatically the semantics from visual data is a real challenge. We describe in this paper how recent work in cognitive vision leads to significative results in activity recognition for visualsurveillance and video monitoring. In particular we present work performed in the domain of video understanding in our PULSAR team at INRIA in Sophia Antipolis. Our main objective is to analyse in real-time video streams captured by static video cameras and to recognize their semantic content. We present a cognitive vision approach mixing 4D computer vision techniques and activity recognition based on a priori knowledge. Applications in visualsurveillance and healthcare monitoring are shown. We conclude by current issues in cognitive vision for activity recognition.

This paper presents the results of ETISEO, a performance evaluation project for video surveillance systems. Many other projects have already evaluated the performance of video surveillance systems, but more on an end-user point of view. ETISEO aims at studying the dependency between algorithms and the video characteristics. Firstly we describe ETISEO methodology which consists in addressing each video processing problem separately. Secondly, we present the main evaluation metrics of ETISEO as well as their benefits, limitations and conditions of use. Finally, we discuss about the contributions of ETISEO to the evaluation community.

In many vision problems, the performance of the seg- mentation step is highly dependent on the algorithm selec- tion and its parametrization. These tasks are tricky and time-consumming. In this paper, we present an approach to perform task-oriented segmentation based on segmenta- tion algorithm parameter tuning and learning techniques. We propose a scheme that, for each segmentation algo- rithm to test, first extracts optimal parameters and second learns the region labeling according to the segmentation task. This supervised approach uses two kinds of ground- truth data: manual region-based segmentations and seman- tic region labels. The first step consists in extracting optimal segmentation algorithm parameters by using a closed-loop optimization procedure, an evaluation metric and ground- truth (manual region segmentations). During the second step, region classifiers are trained based on ground-truth annotations (semantic region labels) to allow segmentation labeling. This knowledge (i.e. optimal parameters and learned region classifiers) is then used to produce an op- timized class-based segmentation of new images. Our main contribution is to propose a methodology to easily set up the segmentation task in vision systems. Our method only requires the user to provide segmentation algorithms and labeled ground-truths. The experiment on a biological ap- plication shows that our optimization scheme is reliable for different state-of-the-art segmentation algorithms. Evalua- tion of results also demonstrates that the optimized class- based segmentation achieves a better level of accuracy than the non-optimized class-based segmentation approach.

As mentioned in many papers, a lot of key parameters of image segmentation algorithms are manually tuned by de- signers. This induces a lack of flexibility of the segmentation step in many vision systems. By a dynamic control of these parameters, results of this crucial step could be drastically improved. We propose a scheme to automatically select segmentation algorithm and tune theirs key parameters thanks to a preliminary supervised learning stage. This paper details this learning approach which is composed by three steps: (1) optimal parameters extraction, (2) algorithm selection learning, and (3) generalization of parametrization learning. The major contribution is twofold: segmentation is adapted to the image to segment, and in the same time, this scheme can be used as a generic framework, independant of any application domain.

This paper presents a distributed multi-camera visual surveillance system for automatic scene interpretation of airport aprons. The system comprises two main modules Scene Tracking and Scene Understanding. The Scene Tracking module is responsible for detecting, tracking and classifying the objects on the apron. The Scene Understanding module performs high level interpretation of the apron activities by applying cognitive spatio-temporal reasoning. The performance of the complete system is demonstrated for a range of representative test scenarios.

We propose a method for discovery of composite events in videos. The algorithm processes a set of primitive events such as simple spatial relations between objects obtained from a tracking system and outputs frequent event patterns which can be interpreted as frequent composite events. We use the APRIORI algorithm from the field of data mining for efficient detection of frequent patterns. We adapt this algorithm to handle temporal uncertainty in the data without losing its computational effectiveness. It is formulated as a generic framework in which the context knowledge is clearly separated from the method in form of a similarity measure for comparison between two video activities and a library of primitive events serving as a basis for the composite events.

This paper presents a complete visual surveillance system for the automatic scene interpretation of airport aprons. The system comprises two modules scene tracking and scene understanding. The scene tracking module, comprising a bottom-up methodology, and the scene understanding module, comprising a video event representation and recognition scheme, have been demonstrated to be a valid approach for apron monitoring.

We present in this paper a real-time system for shape recognition. The proposed system is a video and multi-sensor platform that is able to classify the mobile objects evolving in the scene into several expected categories. The key of the recognition method is to compute mobile object properties thanks to the camera and sensors and then to use Bayesian classifiers. A learning phase based on ground truth data is used to train the Bayesian classifiers. Our recognition method has been integrated into an existing access control device used in public transportation (subway) at RATP (Regie Autonome des Transports Parisiens) to improve safety and comfort, to prevent fraud and to count people for statistical matters. The expected categories in this case are mainly "adult", "child", "suitcase" and "two adults close to each other".

This work presents a new object categorization method and shows how it can be used for image retrieval. Our approach involves machine learning and knowledge representation techniques. A major element of our approach is a visual concept ontology composed of several types of concepts (spatial concepts and relations, color concepts and texture concepts). Visual concepts contained in this ontology can be seen as an intermediate layer between domain knowledge and image processing procedures. Our approach is composed of three phases: (1) a knowledge acquisition phase, (2) a learning phase and (3) a categorization phase. This work is mainly focused on phases (2) and (3). A major issue is the symbol grounding problem which consists of linking meaningfully symbols to sensory information. We propose a solution to this difficult issue by showing how learning techniques can map numerical features to visual concepts.

This paper presents a generic cognitive vision platform for the automatic recognition of natural complex objects. The recognition consists of three steps : image processing for numerical object description, mapping of numerical data into symbolic data and semantic interpretation for object recognition. The focus of this paper is the distributed platform architecture composed of three highly specialized knowledge based systems (KBS). The first KBS is dedicated to semantic interpretation. The second one has to deal with the anchoring of symbolic data into image data. The last KBS is dedicated to intelligent image processing. After a brief overview of the natural object recognition problem, this paper describes the three subcomponents of the platform.

We propose a complete model for the image formation of a 3D microscopic translucent object. This model is essentially composed of 3 steps. The first step is the modeling of an object. We define a 3D distribution of complex indexes of refraction. The second step is based on ray tracing and simulates the light propagation inside the specimen. We compute an light intensity volume. Finally, we simulate a light microscope as a blurring model. It forms the 2D final image. This model of blur approximates the real optical transfer function with 5 physical parameters describing the optical system. We show results of our complete model. The computed images are compared to real images of a microscopic glass sphere.

We propose in this paper an approach for recognizing group of people behaviors using multiple cameras with overlapping FOVs (Field Of View). In this context, Behavior recognition first relies on low level motion detection and frame to frame tracking which generate a graph of mobile objects for each camera. Second, to take advantage of all cameras observing the same scene, a combination mechanism is performed to combine the graphs computed for each camera into a global one. This global graph is then used for long term tracking of groups of people evolving in the scene. Finally, the result of the group tracking is used by a higher level module which recognizes predefined scenarios corresponding to specific group behaviors. This article focuses on the graphs combination mechanism and on the recognition of group behaviors. At the end, results on these two algorithms are described.

We propose a new interpretation platform dedicated to video communication. Basically, a video communication system takes an image flow from a camera and broadcasts it in a computer network. The goal of interpretation is to enable the video communication system to adapt automatically the broadcasted image flow, using image filtering, blurring and zooming. For that we need to detect and track individuals in office scenes, then to understand their behaviour. We focus on a new tracking method based on a 3D model of the scene, on explicit models of individuals and on the computation of several possible paths for each individual. The main issues of the tracking algorithm are presented. Finally we show the results of our algorithm for several sequences illustrating office activities in everyday situations.

This paper presents a method to track multiple nonrigid objects in video sequences. First, we present related works on tracking methods. Second, we describe our proposed approach. We use the notion of target to represent the perception of object motion. To handle the particularities of nonrigid objects we define a target as an individually tracked moving region or as a group of moving regions globally tracked. Then we explain how to compute the trajectory of a target and how to compute the correspondences between known targets and moving regions newly detected. In the case of an ambiguous correspondence we define a compound target to freeze the associations between targets and moving regions until a more accurate information is available. Finally we provide an example to illustrate the way we have implemented the proposed tracking method for video-surveillance applications.

The objective of the PASSWORDS Project is to design and develop a prototype of an intelligent video image analysis system for video surveillance and security applications, based on concrete needs expressed by potential users. The goal of the system developed within PASSWORDS is to detect certain dangerous situations in some scenes (e.g. vandalism in a metro station), providing for example a remote operator with an alarm signal. This paper illustrates the different steps which compose the PASSWORDS system, focusing the attention on the image processing module.

This paper describes a generic framework for the realtime interpretation of video-surveillance scenes. The tool we have built according to this framework performs the understanding process up to the online recognition of ongoing scenario with respect to a library of scenario models dedicated to a particular application. This framework is currently applied for the European Esprit project PASSWORDS, the aim of which is to build a complete system for the automatic analysis and interpretation of video-surveillance image sequences.

This paper deals with the integration of image understanding (IU) programs using a knowledge-based approach. The basic concepts of program integration are discussed, and a simple problem-solving model for program integration is outlined. Two types of reasoning, planning and execution control, are identified. A system developed using this model, called OCAPI (Optimizing, Controlling and Automating the Processing of Images), is introduced. OCAPI is an AI environment in which the reasoning used by the IU specialist is formally represented using frames and production rules. An example of the application developed using OCAPI is presented, and the advantages and shortcomings of this approach are discussed.

Matching is recognized as the most difficult step in stereo vision. Many algorithms have been proposed but none of them is robust enough. The primary reason for this is that one of the most useful cues, namely shape similarity, is not appropriately exploited. Making full use of shape similarity for disambiguation requires determining an appropriate criterion and an adequate implementation. We show that a disparity gradient (DG) limit as small as 0.2 can be used to ensure shape similarity and permits the majority of scene surfaces to be correctly handled. This DG limit is a compromise between the quantity of matches and the quality of matching. The implementation of a DG limit is made effective by explicitly taking into account uncertainty in image point positions. A very effcient voting scheme is proposed. Many tests have been performed, mostly on complex real scenes, but none of the parameters has been adjusted. The results are satisfactory both in terms of the quantity of matches and the correctness of these in normal circumstances. 1

The authors address the problem of road obstacle detection. A sensor composed of an eyesafe laser range finder coupled with a charge coupled device (CCD) camera is proposed. This sensor is mounted in front of a vehicle. The basic idea is to first determine 2-D visual targets in intensity images of the camera. The range finder is then used not only to confirm or reject the real existence of the detected visual targets but also to acquire 3-D information of the confirmed visual targets. The central problem of this strategy is the method of detection of 2-D visual targets from intensity images of a road scene. In the method, line segments are considered as significant features. The concept of a line segment of interest and the concept of a dominant line segment are used. Two-imensional visual targets can be effectively determined with the help of the identification of the dominant line segments in an image. The range finder was used to confirm or reject a 2-D visual target.>

The notions of the apparent boundary and the strict apparent boundary of an object, which provide an automatic approach of boundary detection, are presented. It is shown that the apparent boundary and the strict apparent boundary have the same diameter and the same convex hull as the original object. It is also shown that the strict apparent boundary is weakly externally visible, and is a fixpoint of the two operators that find the apparent boundary and the strict apparent boundary.>

Uses the apparent boundary of an object together with the convex hull to provide an automatic approach for object shape analysis. In particular, the authors present the automatic shape characterization of planktonic foraminifera.>

An approach for scene segmentation based on disparity map is proposed. The basic idea is to group features close in space into objects. The advantages of such an approach are that the search takes place in 2D space, instead of 3D space and that the uncertainty can be taken into account easily and uniformly. The authors show how to express a 3D proximity criterion in terms of disparity and how the problem of uncertainty is coped with efficiently. The result on a traffic sequence is presented.>

The detection of mobile 3-D objects using both stereovision and motion information is considered. First, the authors present an optical flow estimation method based on a multiresolution technique for the matching of contour chain points. Then an algorithm of segmentation of the chains using the continuity of the gradient of the apparent velocity is described. A similar algorithm is developed for stereo vision: the segmentation of the chains is performed in function of their 2-D proximity and the continuity of their disparity. These two segmentations are then used to interpret the scene in terms of isolated rigid or deformable objects. Results are shown for two complex real road scenes.>

We are interested in the semantic interpretation of 3-D data obtained by a stereovision algorithm, in the context of indoor scenes. This paper presents a high level process built to find the main plane structures of the scene and to label them with a semantic description. This process is made up of three stages: first, the sorting and enhancement of the input data (3-D segments), second, the generation of affine planes from the sorted 3-D segments and finally, the semantic description of these planes by a classification expert system using the interpreter CLASSIC. This process has been tested on various stereo images and the quality of the results, in robustness and accuracy, are quite good. The advantages of using a knowledge based system are discussed.