A new technique for unsupervised learning of time series data based on the notion of Granger causality is presented. The technique learns pairs of projections of a multivariate data set such that the resulting components -- "driving" and "driven" -- maximize the strength of the Granger causality between the latent time series (how strongly the past of the driving signal predicts the present of the driven signal). A coordinate descent algorithm that learns pairs of coefficient vectors in an alternating fashion is developed and shown to blindly identify the underlying sources (up to scale) on simulated vector autoregressive (VAR) data. The technique is tested on scalp electroencephalography (EEG) data from a motor imagery experiment where the resulting components lateralize with the side of the cued hand, and also on functional magnetic resonance imaging (fMRI) data, where the recovered components express previously reported resting-state networks.

We hypothesize that certain speaker gestures can convey significant information that are correlated to audience engagement. We propose gesture attributes, derived from speakers' tracked hand motions to automatically quantify these gestures from video. Then, we demonstrate a correlation between gesture attributes and an objective method of measuring audience engagement: electroencephalography (EEG) in the domain of political debates. We collect 47 minutes of EEG recordings from each of 20 subjects watching clips of the 2012 U.S. Presidential debates. The subjects are examined in aggregate and in subgroups according to gender and political affiliation. We find statistically significant correlations between gesture attributes (particularly extremal pose) and our feature of engagement derived from EEG both with and without audio. For some stratifications, the Spearman rank correlation reaches as high as rho = 0.283 with p < 0.05, Bonferroni corrected. From these results, we identify those gestures that can be used to measure engagement, principally those that break habitual gestural patterns.

Our society's information technology advancements have resulted in the increasingly problematic issue of information overload—i.e., we have more access to information than we can possibly process. This is nowhere more apparent than in the volume of imagery and video that we can access on a daily basis—for the general public, availability of YouTube video and Google Images, or for the image analysis professional tasked with searching security video or satellite reconnaissance. Which images to look at and how to ensure we see the images that are of most interest to us, begs the question of whether there are smart ways to triage this volume of imagery. Over the past decade, computer vision research has focused on the issue of ranking and indexing imagery. However, computer vision is limited in its ability to identify interesting imagery, particularly as “interesting” might be defined by an individual. In this paper we describe our efforts in developing brain–computer interfaces (BCIs) which synergistically integrate computer vision and human vision so as to construct a system for image triage. Our approach exploits machine learning for real-time decoding of brain signals which are recorded noninvasively via electroencephalography (EEG). The signals we decode are specific for events related to imagery attracting a user's attention. We describe two architectures we have developed for this type of cortically coupled computer vision and discuss potential applications and challenges for the future.

The minimum variance distortionless response (MVDR) beamformer, also known as Capon's beamformer, is widely studied in the area of speech enhancement. The MVDR beamformer can be used for both speech dereverberation and noise reduction. This paper provides new insights into the MVDR beamformer. Specifically, the local and global behavior of the MVDR beamformer is analyzed and novel forms of the MVDR filter are derived and discussed. In earlier works it was observed that there is a tradeoff between the amount of speech dereverberation and noise reduction when the MVDR beamformer is used. Here, the tradeoff between speech dereverberation and noise reduction is analyzed thoroughly. The local and global behavior, as well as the tradeoff, is analyzed for different noise fields such as, for example, a mixture of coherent and non-coherent noise fields, entirely non-coherent noise fields and diffuse noise fields. It is shown that maximum noise reduction is achieved when the MVDR beamformer is used for noise reduction only. The amount of noise reduction that is sacrificed when complete dereverberation is required depends on the direct-to-reverberation ratio of the acoustic impulse response between the source and the reference microphone. The performance evaluation supports the theoretical analysis and demonstrates the tradeoff between speech dereverberation and noise reduction. When desiring both speech dereverberation and noise reduction, the results also demonstrate that the amount of noise reduction that is sacrificed decreases when the number of microphones increases.

The removal of noise and interference from an array of received signals is a most fundamental problem in signal processing research. To date, many well-known solutions based on second-order statistics (SOS) have been proposed. This paper views the signal enhancement problem as one of maximizing the mutual information between the source signal and array output. It is shown that if the signal and noise are Gaussian, the maximum mutual information estimation (MMIE) solution is not unique but consists of an infinite set of solutions which encompass the SOS-based optimal filters. The application of the MMIE principle to Laplacian signals is then examined by considering the important problem of estimating a speech signal from a set of noisy observations. It is revealed that while speech (well modeled by a Laplacian distribution) possesses higher order statistics (HOS), the well-known SOS-based optimal filters maximize the Laplacian mutual information as well; that is, the Laplacian mutual information differs from the Gaussian mutual information by a single term whose dependence on the beamforming weights is negligible. Simulation results verify these findings.

The acoustic source localization problem has significance for modern intelligent communication systems and future human-computer interaction applications. Although steered-beamforming localization methods perform well in adverse conditions, the algorithms are inherently computationally burdensome due to their need to search the entire location space. This paper presents a computationally- reduced version of the steered response power (SRP) method. The proposed approach is rooted in the inverse mapping of relative delays to candidate source locations, which allows for the transformation of the iterative search from the multidimensional location space to the one-dimensional relative delay space. By subsetting the set of traversed relative delays to only those that experience a high-level of cross-correlation, the computational load is reduced by up to 90 % without incurring a loss in localization accuracy.

From an audio perspective, the present state of teleconferencing technology leaves something to be desired; speaker overlap is one of the causes of this inadequate performance. To that end, this paper presents a frequency-domain implementation of convolutive BSS specifically designed for the nature of the teleconferencing environment. In addition to presenting a novel depermutation scheme, this paper presents a least-squares post-processing scheme, which exploits segments during which only a subset of all speakers are active. Experiments with simulated and real data demonstrate the ability of the proposed methods to provide SIRs at or near that of the adaptive noise cancellation (ANC) solution which is obtained under idealistic assumptions that the ANC filters are adapted with one source being on at a time.

A signal's spectrum is a representation of the signal in terms of elementary basis functions which facilitates the extraction of desired information. For a temporal signal, the spectrum is one-dimensional and expresses the time-domain signal as a linear combination of sinusoidal basis functions. A space-time signal possesses a multidimensional Fourier transform known as the wavenumber-frequency spectrum, which represents the space-time signal as a weighted summation of monochromatic plane waves. The spatial and temporal frequencies are not separable, as spatial frequency is itself a function of the temporal frequency. Thus, it seems natural to analyze and estimate the spatial and temporal frequency components in tandem. It is therefore surprising that conventional spectral estimation methods focus on either the spatial or temporal dimension, without any regard for the other. Spatial spectral estimation is commonly referred to as source localization, as the direction of the wavenumber vector is indeed the direction of propagation. Conventional methods analyze a solely spatial aperture without accounting for the temporal structure of the desired signal. Conversely, temporal spectral estimation is performed using a single sensor, and thus the signal aperture is purely temporal. This paper proposes a spatiotemporal framework for spectral estimation based on the linearly constrained minimum variance (LCMV) beamforming method proposed by Frost in 1972. The aperture consists of an array of sensors, each storing a set of previous temporal samples. It is first shown that by taking into account the temporal structure of the desired signal, the ensuing source location estimate is more robust to the effects of noise and reverberation. Unlike conventional localizers, the LCMV steered beamtemporallyfocuses the array onto the desired signal. The desired signal is modeled by an autoregressive (AR) process, and the resulting AR coefficients are embedded in the linear constraints. As a result, the rate of anomalous estimates is significantly reduced as compared to existing techniques. Moreover, it is then demonstrated that by employing multiple sensors and steering the array to the assumed source location, the estimate of the desired signal's temporal spectrum contains a lesser contribution from the unwanted noise and reverberation.

The estimation of the direction-of-arrival (DOA) of one or more acoustic sources is an area that has generated much interest in recent years, with applications like automatic video camera steering and multi-party stereophonic teleconferencing entering the market. Time-difference-of-arrival (TDOA) based methods compute each relative delay using only two microphones, even though additional microphones are usually available, and thus suffer from the effects of background noise and reverberation. This paper deals with DOA estimation based on spatial spectral estimation, and proposes a novel DOA estimator based on the eigenvalues of the parameterized spatial correlation matrix. Simulation results confirm the ability of the proposed method to provide reliable estimates even in heavily reverberant environments.

Although many microphone-array beamforming algorithms have been developed over the past few decades, most such algorithms so far can only offer limited performance in practical acoustic environments. The reason behind this has not been fully understood and further research on this matter is indispensable. In this paper, we treat a microphone array as a multiple-input multiple-output (MIMO) system and study its signal-enhancement performance. Our major contribution is fourfold. First, we develop a general framework for analyzing performance of beamforming algorithms based on the acoustic MIMO channel impulse responses. Second, we study the bounds for the length of the beamforming filter, which in turn shows the performance bounds of beamforming in terms of speech dereverberation and interference suppression. Third, we address the connection between beamforming and the multiple-input/output inverse theorem (MINT). Finally, we discuss the intrinsic relationships among different classical beamforming techniques and explain, from the channel condition perspective, what the prerequisites are for those techniques to work.

The estimation of the direction-of-arrival (DOA) of one or more acoustic sources is an area that has generated much interest in recent years, with applications like automatic video camera steering and multiparty stereophonic teleconferencing entering the market. DOA estimation algorithms are hindered by the effects of background noise and reverberation. Methods based on the time-differences-of-arrival (TDOA) are commonly used to determine the azimuth angle of arrival of an acoustic source. TDOA-based methods compute each relative delay using only two microphones, even though additional microphones are usually available. This paper deals with DOA estimation based on spatial spectral estimation, and establishes the parameterized spatial correlation matrix as the framework for this class of DOA estimators. This matrix jointly takes into account all pairs of microphones, and is at the heart of several broadband spatial spectral estimators, including steered-response power (SRP) algorithms. This paper reviews and evaluates these broadband spatial spectral estimators, comparing their performance to TDOA-based locators. In addition, an eigenanalysis of the parameterized spatial correlation matrix is performed and reveals that such analysis allows one to estimate the channel attenuation from factors such as uncalibrated microphones. This estimate generalizes the broadband minimum variance spatial spectral estimator to more general signal models. A DOA estimator based on the multichannel cross correlation coefficient (MCCC) is also proposed. The performance of all proposed algorithms is included in the evaluation. It is shown that adding extra microphones helps combat the effects of background noise and reverberation. Furthermore, the link between accurate spatial spectral estimation and corresponding DOA estimation is investigated. The application of the minimum variance and MCCC methods to the spatial spectral estimation problem leads to better resolution than that of the commonly used fixed-weighted SRP spectrum. However, this increased spatial spectral resolution does not always translate to more accurate DOA estimation.

The process of locating an acoustic source given measurements of the sound field at multiple microphones is of significant interest as both a classical array signal processing problem, and more recently, as a solution to the problems of automatic camera steering, teleconferencing, hands-free processing, and others. Despite the proven efficacy of steered-beamformer approaches to localization in harsh conditions, their practical application to real-time settings is hindered by undesirably high computational demands. This paper presents a computationally viable implementation of the steered response power (SRP) source localization method. The conventional approach is generalized by introducing an inverse mapping that maps relative delays to sets of candidate locations. Instead of traversing the three-dimensional location space, the one-dimensional relative delay space is traversed; at each lag, all locations which are inverse mapped by that delay are updated. This means that the computation of the SRP map is no longer performed sequentially in space. Most importantly, by subsetting the space of relative delays to only those that achieve a high level of cross-correlation, the required number of algorithm updates is drastically reduced without compromising localization accuracy. The generalization is scalable in the sense that the level of subsetting is an algorithm parameter. It is shown that this generalization may be viewed as a spatial decomposition of the SRP energy map into weighted basis functions-in this context, it becomes evident that the full SRP search considers all basis functions (even the ones with very low weighting). On the other hand, it is shown that by only including a few basis functions per microphone pair, the SRP map is quite accurately represented. As a result, in a real environment, the proposed generalization achieves virtually the same anomaly rate as the full SRP search while only performing 10% the amount of algorithm updates as the full search.