This paper presents the feasibility of a low-cost video surveillance system that can be used at home to assess the fall risk of older adults. This is of paramount importance since fall is the greatest hazard for older adults. To detect early signs of mobility decline in older adults the system simply detects near-falls with machine learning as part of a fall prevention plan. A One-Class SVM was trained to combine spatiotemporal features from normal activities of daily living. The spatiotemporal features were extracted from a simplified skeleton fitted to the body based on a keypoint RCNN algorithm. Then the system was used to estimate normality scores to identify abnormal events. In practice, a near-fall will trigger a notification to document the fall risk probability. Our experimental results demonstrated that the One-Class SVM could successfully distinguish anomalies (near-falls) with a detection accuracy of 90%, specificity of 87.67% and sensitivity of 93.33% on a dataset of 55 videos (> 16000 frames) of simulated normal and abnormal activities in a realistic apartment-laboratory.

Anomaly detection is a key functionality in various vision systems, such as surveillance and security. In this work, we present a convolutional neural network (CNN) that supports the detection of anomaly, which has not been defined when building the model, at frame level. Our CNN, named SmithNet, is structured to simultaneously learn commonly occurring textures and their corresponding motion. Its architecture is a combination of: 1) an encoder extracting motion-texture coherence from each video frame and 2) two decoders that separately reconstruct the input as well as predict its typical motion from the estimated coherence. We also introduce an encoding block, which is specifically designed for the task of anomaly detection. The optimization is performed on only data of normal events, and the network is expected to determine the ones that are unusual, i.e., have not been seen before. According to the experiments on eight benchmark datasets of different environments with various anomalous events, the performance of our network is competitive or outperforms current state-of-the-art approaches.

Anomaly detection in surveillance videos is currently a challenge because of the diversity of possible events. We propose a deep convolutional neural network (CNN) that addresses this problem by learning a correspondence between common object appearances (e.g. pedestrian, background, tree, etc.) and their associated motions. Our model is designed as a combination of a reconstruction network and an image translation model that share the same encoder. The former sub-network determines the most significant structures that appear in video frames and the latter one attempts to associate motion templates to such structures. The training stage is performed using only videos of normal events and the model is then capable to estimate frame-level scores for an unknown input. The experiments on 6 benchmark datasets demonstrate the competitive performance of the proposed approach with respect to state-of-the-art methods.

In this paper, we propose a method that estimates a gait index for a sequence of skeletons. Our system is a stack of an encoder and a decoder that are formed by Long Short-Term Memories (LSTMs). In the encoding stage, the characteristics of an input are automatically determined and are compressed into a latent space. The decoding stage then attempts to reconstruct the input according to such intermediate representation. The reconstruction error is thus considered as a weak gait index. By combining such weak indices over a long-time movement, our system can provide a good estimation for the gait index. Our experiments on a large dataset (nearly one hundred thousand skeletons) showed that the index given by the proposed method outperformed some recent works on gait analysis.

Photovoltaic panels represent an intermittent way of providing electricity without altering natural resources. Due to its intermittent nature and to meet consumers demand, this energy needs to be stored, for example by using a battery. In this work, an energy model for buildings with battery is developed based on electrical consumption and production data. This model takes into account the depth of discharge, state of charge and efficiency over a cycle of a lithium type battery. Three rule-based strategies are then described. This leads to our optimization problem and proposed algorithm which is compared with two other algorithms. The optimization is applied on two cases, one concerning one day, the other one concerning one week. While the optimization for one day gives quite good results; on one week, the number of variable gives a lot harder problem.

Photovoltaic panels represent an intermittent way of providing electricity without altering natural resources. Due to its intermittent nature and to meet consumers demand, this energy needs to be stored, for example by using a battery. In this work, an energy model for buildings with battery is developed, based on electrical consumption and production data. This model takes into account the depth of discharge, state of charge and efficiency over a cycle of a lithium type battery. Three rule-based strategies are then described. This leads to our optimization problem and proposed algorithm. The optimization is applied on two cases, one concerning a day, the other a concerning one week. While the optimization for one day gives quite good results, on one week, the number of variable leads to a much harder problem. To overpass this problem, a simulation of a real-time system is made. Optimization is conducted over one day and repeated at each step.

This paper adresses the multiple-webcam synchronisation problem for dynamic scenes by proposing a new triangulation method which is invariant to the temporal offset of the webcams. Our method estimates the correct positions of the tracked points without knowing or directly estimating the temporal offset between the cameras. Instead, it relies on the epipolar geometry of the cameras. In the presence of a temporal delay, a tracked point does not lay on its corresponding epipolar line, thereby inducing triangulation errors. By intersecting its motion trajectory with the corresponding epipolar line, an accurate triangulation can be obtained. One major advantage of our approach is that the temporal offset can change throughout the sequence. Temporal offset variations occur frequently when using webcams. The major contribution of this work is making a temporal offset invariant triangulation solution.

Static hand gesture recognition plays an important role in developing a system for human-computer interaction. Besides, such systems can be also used by the deaf community in order to convey information through gestures instead of words. A vision-based processing of hand gesture recognition consists of three main stages: preprocessing, feature extraction and identification. In this paper, the first stage involves following two sub-stages: segmentation which locates hand using color information and extracts its silhouette; separation that separates arm, the part with less information, based on geometrical properties. In the second stage, features which extracted from hand-without-arm are general (ratio of width to height, wrist angle and number of fingers) and detailed (calculated based on fingertips and cross sections) characteristics. Finally, support vector machine model with "max-wins" voting strategy is used to classify the hand gestures. The experiment is conducted on color image dataset of Polish Ministry of Science and Higher Education, with 89.5% classification accuracy.

Automatic facial expression analysis systems are aiming towards the application of computer vision techniques in human computer interaction, emotion analysis, and even medical care via a space mapping between the continuous emotion and a set of discrete expression categories. The main difficulty with these systems is the inherent problem of facial alignment due to person-specific appearance. Beside the facial representation problem, the same displayed facial expression may vary differently across humans; this can be true even for the same person in different contexts. To cope with these variable factors, we introduce the concept of prototype-based model as anchor modeling through a SIFT-flow registration. A set of prototype facial expression models is generated as a reference space of emotions on which face images are projected to generate a set of registered faces. To characterize the facial expression appearance, oriented gradients are processed on each registered image. We obtained the best results 87% with the person–independent evaluation strategy on JAFFE dataset (7-class expression recognition problem), and 83% on the complex setting of the GEMEP-FERA database (5-class problem).

Automatic facial expression analysis is the most commonly studied aspect of behavior understanding and human-computer interface. The main difficulty with facial emotion recognition system is to implement general expression models. The same facial expression may vary differently across humans; this can be true even for the same person when the expression is displayed in different contexts. These factors present a significant challenge for the recognition task. The method we applied, which is reminiscent of the “baseline method”, utilizes dynamic dense appearance descriptors and statistical machine learning techniques. Histograms of oriented gradients (HoG) are used to extract the appearance features by accumulating the gradient magnitudes for a set of orientations in 1-D histograms defined over a size-adaptive dense grid, and Support Vector Machines with Radial Basis Function kernels are the base learners of emotions. The overall classification performance of the emotion detection reached 70% which is better than the 56% accuracy achieved by the “baseline method” presented by the challenge organizers.

Faced with the growing population of seniors, developed countries need to establish new healthcare systems to ensure the safety of elderly people at home. Computer vision provides a promising solution to analyze personal behavior and detect certain unusual events such as falls. In this paper, a new method is proposed to detect falls by analyzing human shape deformation during a video sequence. A shape matching technique is used to track the person's silhouette along the video sequence. The shape deformation is then quantified from these silhouettes based on shape analysis methods. Finally, falls are detected from normal activities using a Gaussian mixture model. This paper has been conducted on a realistic data set of daily activities and simulated falls, and gives very good results (as low as 0% error with a multi-camera setup) compared with other common image processing methods.

According to the demographic evolution in industrialized countries, more and more elderly people will experience falls at home and will require emergency services. The main problem comes from fall-prone elderly living alone at home. To resolve this lack of safety, we propose a new method to detect falls at home, based on a multiple-cameras network for reconstructing the 3-D shape of people. Fall events are detected by analyzing the volume distribution along the vertical axis, and an alarm is triggered when the major part of this distribution is abnormally near the floor during a predefined period of time, which implies that a person has fallen on the floor. This method was validated with videos of a healthy subject who performed 24 realistic scenarios showing 22 fall events and 24 cofounding events (11 crouching position, 9 sitting position, and 4 lying on a sofa position) under several camera configurations, and achieved 99.7% sensitivity and specificity or better with four cameras or more. A real-time implementation using a graphic processing unit (GPU) reached 10 frames per second (fps) with 8 cameras, and 16 fps with 3 cameras.

The goal of this work is to perform a segmentation of the intimamedia thickness (IMT) of carotid arteries in view of computing various dynamical properties of that tissue, such as the elasticity distribution (elastogram). The echogenicity of a region of interest comprising the intima-media layers, the lumen, and the adventitia in an ultrasonic B-mode image is modeled by a mixture of three Nakagami distributions. In a first step, we compute the maximum a posteriori estimator of the proposed model, using the expectation maximization (EM) algorithm. We then compute the optimal segmentation based on the estimated distributions as well as a statistical prior for disease-free IMT using a variant of the exploration/selection (ES) algorithm. Convergence of the ES algorithm to the optimal solution is assured asymptotically and is independent of the initial solution. In particular, our method is well suited to a semi-automatic context that requires minimal manual initialization. Tests of the proposed method on 30 sequences of ultrasonic B-mode images of presumably disease-free control subjects are reported. They suggest that the semi-automatic segmentations obtained by the proposed method are within the variability of the manual segmentations of two experts.

3D Brain SPECT imagery is a well established functional imaging method which has become a great help to physicians in the diagnosis of several neurological and cerebrovascular diseases. However, mainly due to the effects of attenuation and the scattering of emitted photons, inherent to this imaging process, 3D SPECT images are generally blurred and exhibit poor spatial resolution. This leads to substantial errors in measurements of regional brain blood flow, and therefore in the estimations of brain activity. In order to improve the resolution of these images and then to facilitate their interpretation, we herein propose an original extension of the NAS-RIF (Recursive Inverse Filtering) deconvolution technique proposed by Kundur and Hatzinakos [1]. The proposed extension allows to efficiently integrate, in the deconvolution process, a set of soft constraints given by a probabilistic MRI atlas containing experts's prior knowledge about the spatial localization of the different brain structures (or tissue classes). This extension has three interesting properties ; first it allows to exploit (or fuse) reliable anatomical and (high resolution) geometrical information extracted horn a probabilistic 3D MRI atlas. Second, it allows to incorporate, into the NAS-RIF method, a regularization term which efficiently stabilizes the inverse solution. Third and contrary to multi-modal restoration techniques, it does not require a MRI scan of the patient. This method has been successfully tested on numerous real brain SPECT images (of different patients suffering from epilepsy), yielding promising restoration results.

During an intravascular ultrasound (IVUS) intervention, a catheter with an ultrasound transducer is introduced in the body through a blood vessel, and then, pulled back to image a sequence of vessel cross sections. Unfortunately, there is no 3-D information about the position and orientation of these cross-section planes, which makes them less informative. To position the IVUS images in space, some researchers have proposed complex stereoscopic procedures relying on biplane angiography to get two X-ray image sequences of the IVUS transducer trajectory along the catheter. To simplify this procedure, we and others have elaborated algorithms to recover the transducer 3-D trajectory with only a single view X-ray image sequence. In this paper, we present an improved method that provides both automated 2-D and 3-D transducer tracking based on pullback speed as a priori information. The proposed algorithm is robust to erratic pullback speed and is more accurate than the previous single-plane 3-D tracking methods.

Nowadays, brain 3D SPECT is a well established functional imaging method that is widely used in clinical settings for the assessment of neurological and cerebrovascular diseases. However, due to the scattering of the emitted photons, inherent to this imaging process, brain 3D SPECT images exhibit poor spatial and inter-slice resolution. More precisely, SPECT images are blurred, leading to substantial errors in measurement of regional brain activity and making difficult and subjective, a reliable and accurate diagnosis by the nuclear physician. In order to improve the resolution of these images and then to facilitate their interpretation, we herein propose an original extension of the NAS-RIF deconvolution technique of Kundur and Hatzinakos. The proposed extension has two interesting properties; it allows to exploit or fuse anatomical and geometrical information extracted from a high resolution anatomical magnetic resonance (MR) image and also to efficiently incorporate, into the NAS-RIF method, a regularization term to stabilize the inverse solution. In our application, this anatomical-based regularization term exploits the result of an unsupervised Markovian segmentation obtained after a preliminary registration step between the MR and SPECT data volume coming from a same patient. This method has been successfully tested on numerous pairs of brain MR and SPECT images of different patients, yielding very promising restoration results.

Intravascular ultrasound (IVUS) is a catheter based medical imaging technique particularly useful for studying atherosclerotic disease. It produces cross-sectional images of blood vessels that provide quantitative assessment of the vascular wall, information about the nature of atherosclerotic lesions as well as plaque shape and size. Automatic processing of large IVUS data sets represents an important challenge due to ultrasound speckle, catheter artifacts or calcification shadows. A new three-dimensional (3-D) IVUS segmentation model, that is based on the fast-marching method and uses gray level probability density functions (PDFs) of the vessel wall structures, was developed. The gray level distribution of the whole IVUS pullback was modeled with a mixture of Rayleigh PDFs. With multiple interface fast-marching segmentation, the lumen, intima plus plaque structure, and media layers of the vessel wall were computed simultaneously. The PDF-based fast-marching was applied to 9 in vivo IVUS pullbacks of superficial femoral arteries and to a simulated IVUS pullback. Accurate results were obtained on simulated data with average point to point distances between detected vessel wall borders and ground truth <0.072 mm. On in vivo IVUS, a good overall performance was obtained with average distance between segmentation results and manually traced contours <0.16 mm. Moreover, the worst point to point variation between detected and manually traced contours stayed low with Hausdorff distances <0.40 mm, indicating a good performance in regions lacking information or containing artifacts. In conclusion, segmentation results demonstrated the potential of gray level PDF and fast-marching methods in 3-D IVUS image processing.

This paper presents an original method for three-dimensional elastic registration of multimodal images. We propose to make use of a scheme that iterates between correcting for intensity differences between images and performing standard monomodal registration. The core of our contribution resides in providing a method that finds the transformation that maps the intensities of one image to those of another. It makes the assumption that there are at most two functional dependencies between the intensities of structures present in the images to register, and relies on robust estimation techniques to evaluate these functions. We provide results showing successful registration between several imaging modalities involving segmentations, T1 magnetic resonance (MR), T2 MR, proton density (PD) MR and computed tomography (CT). We also argue that our intensity modeling may be more appropriate than mutual information (MI) in the context of evaluating high-dimensional deformations, as it puts more constraints on the parameters to be estimated and, thus, permits a better search of the parameter space.

We present a new method to shape-based segmentation of deformable anatomical structures in medical images and validate this approach by detecting and tracking the endocardial border in an echographic image sequence. To this end, a global prior knowledge of the endocardial contour is captured by a prototype template with a set of admissible deformations to take into account its inherent natural variability over time. In this approach, the data likelihood model rely on an accurate statistical modeling of the grey level distribution of each class present in the image. The parameters of this distribution mixture are given by a preliminary estimation step which takes into account the distribution shape of each class. Then the tracking problem is stated in a Bayesian framework where it ends up as an optimization problem. This one is then efficiently solved by a genetic algorithm combined with a steepest ascent procedure. This technique has been successfully applied on synthetic images and on a real echocardiographic image sequence.

We propose anew rectification method for aligning epipolar lines of a pair of stereo images taken under any camera geometry. It effectively remaps both images onto the surface of a cylinder instead of a plane, which is used in common rectification methods. For a large set of camera motions, remapping to a plane has the drawback of creating rectified images that are potentially infinitely large and presents a loss of pixel information along epipolar lines. In contrast, cylindrical rectification guarantees that the rectified images are bounded for all possible camera motions and minimizes the loss of pixel information along epipolar line. The processes (e.g., stereo matching, etc.) subsequently applied to the rectified images are thus more accurate and general since they can accommodate any camera geometry.

A theoretical model was previously developed to evaluate the relationship between the dynamics of ultrasonic speckle and its underlying tissue. The model is divided into an instrumental part represented by the point spread function (in the far field) of the ultrasonic apparatus and a moving tissue component described by a collection of scatterers. By computing the convolution of these terms and then the envelope, one obtains a simulated ultrasonic speckle pattern sequence which shows speckle motions closely linked to the tissue dynamics when small motion amplitudes are involved. Here, a theoretical study of the correlation between various linear transformations of the tissue and the corresponding ultrasonic speckle motions is performed, based on a 2D extension of the envelope cross-correlation analysis of a narrow-band Gaussian noise. In the linear scan case, obviously, tissue translation generates an identical speckle translation. However, tissue/speckle motion correlation decreases with increasing rotation and/or biaxial deformation, lateral deformation (perpendicular to the beam propagation axis) being much less sensitive. With respect to the transducer frequency, the rotation and the axial deformation of the tissue show a better relationship with their respective speckle motion at lower frequencies while lateral deformation correlation is independent of the pulse frequency. With respect to beam (pulse) size parameters, tissue/speckle correlation decreases with rotation when a wide ultrasonic beam is used while the axial deformation correlation decreases with the axial duration of the pulse. This study sets the ground for the development of an ultrasonic strain gauge particularly useful for the assessment of biomechanical soft tissue and fluid flow properties based on speckle tracking.