This paper presents a smart surveillance system named CASSANDRA, aimed at detecting instances of aggressive human behavior in public environments. A distinguishing aspect of CASSANDRA is the exploitation of complementary audio and video cues to disambiguate scene activity in real-life environments. From the video side, the system uses overlapping cameras to track persons in 3D and to extract features regarding the limb motion relative to the torso. From the audio side, it classifies instances of speech, screaming, singing, and kicking-object. The audio and video cues are fused with contextual cues (interaction, auxiliary objects); a Dynamic Bayesian Network (DBN) produces an estimate of the ambient aggression level. Our prototype system is validated on a realistic set of scenarios performed by professional actors at an actual train station to ensure a realistic audio and video noise setting.

This article compares methods for the conversion of time-series into a spectro-temporal representation. These methods are designed based on a resemblance with the auditory processing of sound in the inner ear, or on mathematical principles related to, for example, Fourier analysis. This study provides a comparison between several of these methods. Two tests were devised for this comparison: one based on susceptibility to noise and one on the expression of spectro-temporal detail. While some methods produced good results on one of the two tests, others produced good results on both. Overall the transmission line model using an impedance function suggested by Zweig [1] provided the best results, though not significantly. Also a larger computational load may hinder application in some domains. The gammatone filterbank and straightforward spectrogram provide good alternatives with less computational load.

This article compares methods for the conversion of timeseries into a spectro-temporal representation. These methods are designed based on a resemblance with the auditory processing of sound in the mammalian inner ear, or on mathematical principles related to, for example, Fourier analysis. This study provides a comparison between several of these methods. Two tests were devised for this comparison: one based on susceptibility to noise and one on the expression of spectro-temporal detail. These two aspects were considered of importance for real world applications. While some methods produced good results on one of the two tests, others produced good results on both. Overall the transmission line model using an impedance function suggested by Zweig (“Finding the impedance of the organ of Corti,” J. Acoust. Soc. Amer., vol. 89, no. 3, pp. 1229-1254, 1991) provided the best results, though not significantly. Also a larger computational load may hinder application in some domains. The gammatone filterbank and straightforward spectrogram provide good alternatives with less computational load. The introduction of nonlinearity was shown to deteriorate performance on both tests, in both the filterbank and in the transmission line model.

This paper presents a smart surveillance system named CASSANDRA, aimed at detecting instances of aggressive human behavior in public environments. A distinguishing aspect of CASSANDRA is the exploitation of the complimentary nature of audio and video sensing to disambiguate scene activity in real-life, noisy and dynamic environments. At the lower level, independent analysis of the audio and video streams yields intermediate descriptors of a scene like: "scream", "passing train" or "articulation energy". At the higher level, a Dynamic Bayesian Network is used as a fusion mechanism that produces an aggregate aggression indication for the current scene. Our prototype system is validated on a set of scenarios performed by professional actors at an actual train station to ensure a realistic audio and video noise setting.