Neural representations estimated from functional MRI (fMRI) responses to natural sounds in non-primary auditory cortical areas resemble those in intermediate layers of deep neural networks (DNNs) trained to recognize sounds. However, the nature of these representations remains poorly understood. In the current study, a convolutional DNN (YAMNet), pre-trained to map sound spectrograms to semantic categories, is used as a computer simulation of the human brain’s processing of natural sounds. A novel sound data set is introduced and employed to test the hypothesis that sound-to-event DNNs represent basic mechanisms of sound generation (here, human actions ) and physical properties of the sources (here, object materials ) in their intermediate layers. Systematic changes to those latent representations are made with the help of a disentangling flow model . The manipulations are shown to cause a predictable effect on DNN’s semantic output. By demonstrating this mechanism in silico , the current study paves the way for neuroscientific experiments aiming to verify it in vivo . Code available at https://github.com/TimHenry1995/LatentAudio .

Driven by the appeal of real-world applicable models, we investigate how temporal and spatial occlusion affect sign language recognition. Utilizing only a crop of the hands and pose flow, we maintain accuracies comparable to an I3D baseline for the WLASL dataset using a video transformer model (VTN), implying that hand crops might contain enough information for accurate prediction. Moreover, we find that a crop of only the right hand provides enough data to train an accurate model, achieving results of 0.2% less than the baseline for AUTSL and 4.7% less across all WLASL datasets. Sampling a video every fifth frame achieves comparative results to baseline, with 8 frame sequences performing better for AUTSL (0.4% less than baseline) and 16 frames performing better for WLASL (0.2% for WLASL 100 and 300). Our results indicate the feasibility of utilizing less information for sign language recognition, however more research is necessary to apply these findings in real-world scenarios.

This paper presents the novel use of convolutional neural network (CNN)-based machine learning models for remotely detecting and monitoring retrogressive thaw slumps (RTS) in high latitude northern permafrost using open-source Sentinel-2 satellite data. RTS are indicative of rapid permafrost thaw (RPT), the accelerated release of greenhouse gases (GHG) and potentially runaway changes in the cryosphere. Attempts to quantify GHG emissions from RTS are inhibited by a lack of information on RTS incidence and area affected. We show that site-specific CNN models can be used to produce time series data on rapid RTS development that allow for the approximation of associated GHG emissions. For the sites assessed we achieve good model precision, recall and F1 values of > 0.8. The short time series studied so far do not reveal clear trends in RTS development. These limitations arise from the low resolution of Sentinel-2 data (10 m) and limited availability and diversity of validated training data. The capability shown here is the first step towards achieving automated monitoring of rapid environmental change in permafrost using satellite data. This work highlights the need for ready access to open-source high resolution satellite data and permafrost field data if the potential of such approaches is to be fully realized.

A new approach for unknown event detection in videos with dense motion, such as crowds or dynamic textures, is developed, without requiring the estimation of optical flow, with no prior knowledge about normal or abnormal events, and with no training. The proposed method directly extracts motion statistics from the phase of the video's Fourier transform and detects changes in them, and in the video, by applying sequential statistical change detection theory. Focus is placed on the motion component, as videos of densely moving entities, such as temporal textures and crowds, often have a very similar appearance, but different dynamic features. Experiments with synthetically generated datasets demonstrate the method's operation under various conditions, while experiments on a recently introduced crowd dataset show that it succeeds at detecting new events in videos of crowds, with no training, and no prior knowledge of the location of new events in space and time.

Ever since the emergence of digitization, the term multimedia has been used to represent a combination of different kinds of media types, such images, audio, and videos. As new sensing technologies emerge and are now becoming omnipresent in daily lives, the definition, role and significance of multimedia is changing. Multimedia now represents the means for communicating, cooperating, and also for monitoring numerous aspects of daily life, at various levels of granularity and application, ranging from personal to societal. With this shift, we have since moved from comprehending single media and its state toward comprehending media in terms of its use context. Multimedia is thus no longer confined to documentation and preservation, entertainment or personal media collections; rather, it has become an integral part of the tools and systems that are providing solutions to today's societal challenges-including challenges related to health care and personal health , aging, education, societal participation, sustainable energy, and intelligent transportation. Multimedia has thus evolved into a core enabler for future interactive and cooperative applications at the heart of society. In this workshop we explore the relevance and contribution of multimedia to health care and personal media.

Combining multimodal concept streams from heterogeneous sensors is a problem superficially explored for activity recognition. Most studies explore simple sensors in nearly perfect conditions, where temporal synchronization is guaranteed. Sophisticated fusion schemes adopt problem-specific graphical representations of events that are generally deeply linked with their training data and focused on a single sensor. This paper proposes a hybrid framework between knowledge-driven and probabilistic-driven methods for event representation and recognition. It separates semantic modeling from raw sensor data by using an intermediate semantic representation, namely concepts. It introduces an algorithm for sensor alignment that uses concept similarity as a surrogate for the inaccurate temporal information of real life scenarios. Finally, it proposes the combined use of an ontology language, to overcome the rigidity of previous approaches at model definition, and a probabilistic interpretation for ontological models, which equips the framework with a mechanism to handle noisy and ambiguous concept observations, an ability that most knowledge-driven methods lack. We evaluate our contributions in multimodal recordings of elderly people carrying out IADLs. Results demonstrated that the proposed framework outperforms baseline methods both in event recognition performance and in delimiting the temporal boundaries of event instances.

• Depth descriptors (HOSNP) achieves high acc. rate on location based ADLs such Hand-Shake/ Read Paper • Motion descriptors (MBH /HOF) achieve high accuracy rate for ADLs that feature significant motion such as: Drink, Eat, but at a much higher computational cost. • Appearance descriptors (HOG) give higher recognition rates for static ADLs such as: Read, Talk and discriminate between activities with similar motion : Drink/Eat and Prepare Snack/Prepare Drink Appearance and Depth for Rapid Human Activity Recognition in Real Applications

Camera motion is often present in videos, due to shaking, jitter or ego-motion. This creates the need for a reliable motion compensation algorithm that will lead to accurate spatial activity localization and activity recognition. We detect background areas using SLIC superpixels and multiple homographies, and introduce motionplanes, defined as background areas that are undergoing diverse camera motions. Dense trajectories are sampled from these regions and histograms of appearance and motion are extracted to build a representation scheme to be used for activity recognition. Experiments on two publicly available unconstrained video datasets of human activities recorded with a moving camera show that our algorithm not only outperforms most activity localization state-of-the-art algorithms but also leads to high activity recognition accuracy rates.

In this work, we propose a computationally efficient method for the recognition of human activities of daily living. Our method uses trajectories of tracked visual features extracted on dense grids and performs recognition via Support Vector Machines (SVMs). In contrast to State-of-the-Art approaches, which are based on dense optical flow (OF), we use fast block matching motion estimation, resulting in increased computational efficiency, with minimal loss in terms of recognition accuracy. To prove the effectiveness of our approach, we have conducted experiments on benchmark datasets of videos of human activities of daily living, demonstrating the trade-offs between recognition accuracy and computational efficiency.

This paper focuses on detecting and localizing anomalous events in videos of crowded scenes, i.e., divergences from a dominant pattern. Both motion and appearance information are considered, so as to robustly distinguish different kinds of anomalies, for a wide range of scenarios. A newly introduced concept based on swarm theory, histograms of oriented swarms (HOS), is applied to capture the dynamics of crowded environments. HOS, together with the well-known histograms of oriented gradients, are combined to build a descriptor that effectively characterizes each scene. These appearance and motion features are only extracted within spatiotemporal volumes of moving pixels to ensure robustness to local noise, increase accuracy in the detection of local, nondominant anomalies, and achieve a lower computational cost. Experiments on benchmark data sets containing various situations with human crowds, as well as on traffic data, led to results that surpassed the current state of the art (SoA), confirming the method's efficacy and generality. Finally, the experiments show that our approach achieves significantly higher accuracy, especially for pixel-level event detection compared to SoA methods, at a low computational cost.

In this work we propose a novel approach to the detection of anomalous events occurring in crowded scenes. Swarm theory is applied for the creation of a motion feature first introduced in this work, the Histograms of Oriented Swarm Accelerations (HOSA), which are shown to effectively capture a scene's motion dynamics. The HOSA, together with the well known Histograms of Oriented Gradients (HOGs) describing appearance, are combined to provide a final descriptor based on both motion and appearance, to effectively characterize a crowded scene. Appearance and motion features are only extracted within spatiotemporal volumes of moving pixels (regions of interest) to ensure robustness to local noise and allow the detection of anomalies occurring only in a small region of the frame. Experiments and comparisons with the State of the Art (SoA) on a variety of benchmark datasets demonstrate the effectiveness of the proposed method, its flexibility and applicability to different crowd environments, and its superiority over currently existing approaches.

Healthcare systems are depending on increasingly sophisticated and ubiquitous technology, while telehealth is rapidly gaining importance with the advent of low-cost and effective technological solutions in medicine. The increase in the worldwide elderly population and the burden this is inflicting upon the workforce, societies and economies are making remote care and independent living at home a necessity. MIIRH is the first workshop on multimedia analysis for remote care of and assisted living solutions which enable people that are incapacitated in some regard to continue living independently at home and remain active members of society. The topics addressed in MIIRH are extremely timely, as multitudes of cost-effective and high quality care solutions are already being developed and used, rendering the examination of new medical, healthcare paradigms an absolute necessity.

The automated analysis of crowd behavior from videos has been a rather challenging problem to address due to the complexity and density of the motion, occlusions and local noise. A novel approach for the fast and reliable detection and characterization of abnormal events in crowd motions is proposed, based on particle advection and accurate optical flow estimation. Experiments on benchmark datasets show that changes are detected reliably and faster than existing methods. Also, regions of change are localized spatially, and the events occurring in the video are characterized with accuracy.

This paper presents a new method for human action recognition which exploits advantages of both trajectory and space-time based approaches in order to identify action patterns in given sequences. Videos with both a static and moving camera can be tackled, where camera motion effects are overcome via motion compensation. Only pixels undergoing changing motion, found by extracting motion boundary-based activity areas, are processed in order to introduce robustness to camera motion and reduce computational complexity. In these regions, densely sampled grid points on multiple scales are tracked using a KLT tracker, leading to dense multi-scale trajectories, on which HOGHOF descriptors are estimated. The length of each trajectory is determined by detecting changes in the tracked points' motion or appearance using sequential change detection techniques, namely the CUSUM approach. A vocabulary is created for each video's features using Hierarchical K-means, and the resulting fast search trees are used to describe the actions in the videos. SVMs are used for classification, using a kernel based on the similarity scores between training and testing videos. Experiments are carried out with new and challenging datasets for which the proposed method is shown to lead to recognition results that are comparable to or better than existing state of the art methods.

In this paper, an approach to semantic video analysis that is based on the statistical processing and representation of the motion signal is presented. Overall, the examined video is temporally segmented into shots and for every resulting shot appropriate motion features are extracted; using these, hidden Markov models (HMMs) are employed for performing the association of each shot with one of the semantic classes that are of interest. The novel contributions of this paper lie in the areas of motion information processing and representation. Regarding the motion information processing, the kurtosis of the optical flow motion estimates is calculated for identifying which motion values originate from true motion rather than measurement noise. Additionally, unlike the majority of the approaches of the relevant literature that are mainly limited to global- or camera-level motion representations, a new representation for providing local-level motion information to HMMs is also presented. It focuses only on the pixels where true motion is observed. For the selected pixels, energy distribution-related information, as well as a complementary set of features that highlight particular spatial attributes of the motion signal, are extracted. Experimental results, as well as comparative evaluation, from the application of the proposed approach in the domains ofTennis,NewsandVolleyballbroadcast video, andHuman Actionvideo demonstrate the efficiency of the proposed method.

A robust, theoretically founded approach for the extraction of temporal templates corresponding to areas of motion in video, is presented. Higher order statistics (kurtosis) are employed to extract activity areas, i.e., binary masks indicating which pixels in a video are active. The application of the kurtosis on illumination changes modeled as Gaussians and mixture of Gaussians is shown to be sensitive to outliers for both models, thus correctly localizing active pixels. Activity areas are compared to existing, difference-based temporal templates, known as motion energy images, and the robustness of both categories of temporal templates to additive noise is analyzed theoretically. Experiments with numerous real videos with additive noise, both indoors and outdoors, are conducted to compare the robustness of the activity areas and motion energy images, and their temporal extensions, the activity history areas, and motion history images. As expected from the theoretical analysis, the kurtosis-based activity areas prove to be more robust than the difference-based templates. Challenging videos containing occlusions, varying backgrounds, and shadows are also examined, and it is shown that the proposed approach outperforms the difference-based method for these cases, as well, consistently providing reliable localization of activity under a wide range of difficult circumstances. The proposed approach provides good results at a very low computational cost, and without requiring prior knowledge about the scene, nor training of any kind.

This work presents a novel approach to the extraction of trajectories in video. It has the advantage of simultaneously processing all spatiotemporal information, i.e. all the video frames, and thus overcoming disadvantages of local approaches. A projection of the video frames over time is used to create a frequency modulated signal, which is then processed with the continuous wavelet transform (CWT). The CWT's power is concentrated around the prominent time-varying frequencies, which are proportional to the object trajectory. Consequently it can extract the time-varying motion trajectories, which can be used in future extensions to fully characterize the type motion taking place. This approach is robust to local measurement noise and occlusions, since it processes the available data in a global, integrated manner. Experiments take place with synthetic and real sequences demonstrate the capabilities of this approach.

Recently, advanced video processing systems have been developed for numerous applications, such as surveillance, tracking, monitoring, object and event detection. The large size of this data and the wide range of detectable events or objects render the automation of various video processing procedures necessary. The proposed system first finds the pixels where activities occur by processing the higher order statistics of luminance changes between accumulated video frames. Once these pixels are extracted, sequential likelihood ratio based change detection techniques are applied to find at which frames activities occur. The resulting system is novel, as it can localize actions both in space and in time, in a theoretically sound manner. Testing takes place with real video sequences, to demonstrate its effectiveness.

A video containing multiple objects undergoing independent translational and rotational motions is analyzed through a combination of spatial- and frequency-domain representations. The Fourier transform of the sequence is used to estimate the multiple translations and rotations in a computationally efficient manner, which is also robust to local inaccuracies and global illumination changes. A novel algorithm is presented for the simultaneous extraction of all objects undergoing translation and the background via a least squares technique that takes place entirely in the Fourier domain. Spatial information is combined with the frequency domain object extraction results, to further refine them. For the case of rotational or combined, rotational and translational motions, the moving objects are segmented using purely spatial information. We show that the combined analysis takes advantage of the strengths of both representations, by providing reliable and computationally efficient motion estimates and object segmentation. The proposed algorithm is shown to be robust to local noise and occlusion, because of its global nature. Experiments are performed on synthetic and real video sequences to demonstrate the capabilities of our approach.

In recent years, digital multimedia technologies have evolved significantly, and are finding numerous applications, over the internet, and even over mobile networks. Thus, the video processing community has started focusing more intensively on the extraction of higher level information from multimedia data. This paper proposes a novel two-stage video processing system that aims to segment and extract semantically meaningful information, which can help achieve higher level interpretation of video. The flow fields present in the video are accumulated over several frames and their statistics are processed to derive an "activity area", that is characteristic of the type of events taking place. The color information complements the motion data, and is used for the accurate segmentation of the moving entities in each frame. The joint use of the activity area and accurate segmentation can serve as a first step to the further semantic interpretation of the video, including the recognition and accurate localization of moving objects of interest. We present experiments that demonstrate the effectiveness of our method for real videos.

Vast amounts of digital multimedia data are being produced and distributed today, so methods for the efficient and reliable extraction of information from video data are becoming necessary. We present a novel motion segmentation algorithm, which accurately extracts moving objects from a video, and also provides a likelihood map, for each object pixel assignment. The flow is estimated, and accumulated over several frames, to give action masks. Color segmentation clusters regions of similar color in each frame. A novel, likelihood ratio-based method for the statistical comparison of color layers in the regions of activity and the background is presented and compared with an Earth mover's distance-based approach. Our method also gives the likelihood with which each pixel is assigned to a moving object in each frame. Experiments with real sequences illustrate the advantages of our method, namely that it gives overall more reliable results, and also provides the likelihood map for the segmented object.

The analysis of periodic or repetitive motions is useful in many applications, such as the recognition and classification of human and animal activities. Existing methods for the analysis of periodic motions first extract motion trajectories using spatial information and then determine if they are periodic. These approaches are mostly based on feature matching or spatial correlation, which are often infeasible, unreliable, or computationally demanding. In this paper, we present a new approach, based on the time-frequency analysis of the video sequence as a whole. Multiple periodic trajectories are extracted and their periods are estimated simultaneously. The objects that are moving in a periodic manner are extracted using the spatial domain information. Experiments with synthetic and real sequences display the capabilities of this approach.

A video containing multiple objects in rotational and translational motion is analyzed through a combination of spatial and frequency domain representations. It is argued that the combined analysis can take advantage of the strengths of both representations. Initial estimates of constant, as well as time-varying, translation and rotation velocities are obtained from frequency analysis. Improved motion estimates and motion segmentation for the case of translation are achieved by integrating spatial and Fourier domain information. For combined rotational and translational motions, the frequency representation is used for motion estimation, but only spatial information can be used to separate and extract the independently moving objects. The proposed algorithms are tested on synthetic and real videos.

This paper addresses issues that arise in copyright protection systems of digital images, which employ blind watermark verification structures in the discrete cosine transform (DCT) domain. First, we observe that statistical distributions with heavy algebraic tails, such as the alpha-stable family, are in many cases more accurate modeling tools for the DCT coefficients of JPEG-analyzed images than families with exponential tails such as the generalized Gaussian. Motivated by our modeling results, we then design a new processor for blind watermark detection using the Cauchy member of the alpha-stable family. The Cauchy distribution is chosen because it is the only non-Gaussian symmetric alpha-stable distribution that exists in closed form and also because it leads to the design of a nearly optimum detector with robust detection performance. We analyze the performance of the new detector in terms of the associated probabilities of detection and false alarm and we compare it to the performance of the generalized Gaussian detector by performing experiments with various test images.

We propose a blind image watermarking algorithm that has three main features: (1) it satisfies an undetectability constraint with respect to a statistical image model, (2) it rejects host-signal interference and (3) it is robust against additive and multiplicative noise, filtering, cropping, and random bending.

The widespread use of digital multimedia data has increased the need for effective means of copyright protection. Watermarking has attracted much attention, as it allows the embedding of a signature in a digital document in an imperceptible manner. In practice, watermarking is subject to various attacks, intentional or unintentional, which degrade the embedded information, rendering it more difficult to detect. A very common, but not malicious, attack is quantization, which is unavoidable for the compression and transmission of digital data. The effect of quantization attacks on the nearly optimal Cauchy watermark detector is examined in this paper. The quantization effects on this detection scheme are examined theoretically, by treating the watermark as a dither signal. The theory of dithered quantizers is used in order to correctly analyze the effect of quantization attacks on the Cauchy watermark detector. The theoretical results are verified by experiments that demonstrate the quantization effects on the detection and error probabilities of the Cauchy detection scheme.

The issue of copyright protection of digital multimedia data has attracted a lot of attention during the last decade. An efficient copyright protection method that has been gaining popularity is watermarking, i.e., the embedding of a signature in a digital document that can be detected only by its rightful owner. Watermarks are usually blindly detected using correlating structures, which would be optimal in the case of Gaussian data. However, in the case of DCT-domain image watermarking, the data is more heavy-tailed and the correlator is clearly suboptimal. Nonlinear receivers have been shown to be particularly well suited for the detection of weak signals in heavy-tailed noise, as they are locally optimal. This motivates the use of the Gaussian-tailed zero-memory nonlinearity, as well as the locally optimal Cauchy nonlinearity for the detection of watermarks in DCT transformed images. We analyze the performance of these schemes theoretically and compare it to that of the traditionally used Gaussian correlator, but also to the recently proposed generalized Gaussian detector, which outperforms the correlator. The theoretical analysis and the actual performance of these systems is assessed through experiments, which verify the theoretical analysis and also justify the use of nonlinear structures for watermark detection. The performance of the correlator and the nonlinear detectors in the presence of quantization is also analyzed, using results from dither theory, and also verified experimentally.

This paper studies the effects of desynchronization attacks such as delay and warping on the performance of blind spread-spectrum watermark detection systems. The host signal is modeled as a colored Gaussian signal. Evaluation of the optimal likelihood ratio test is often computationally expensive, so as a practical alternative, we propose a family of quadratic detectors and construct the detector and family of watermarks that maximize the deflection criterion. Experiments are carried out to verify the suitability of the deflection as a performance index. Substantial improvements over conventional watermark designs are demonstrated.