Active speaker detection is an important component in video analysis algorithms for applications such as speaker diarization, video re-targeting for meetings, speech enhancement, and human-robot interaction. The absence of a large, carefully labeled audio-visual active speaker dataset has limited evaluation in terms of data diversity, environments, and accuracy. In this paper, we present the AVA Active Speaker detection dataset (AVA-ActiveSpeaker) which has been publicly released to facilitate algorithm development and comparison. It contains temporally labeled face tracks in videos, where each face instance is labeled as speaking or not, and whether the speech is audible. The dataset contains about 3.65 million human labeled frames spanning 38.5 hours. We also introduce a state-of-the-art, jointly trained audio-visual model for real-time active speaker detection and compare several variants. The evaluation clearly demonstrates a significant gain due to audio-visual modeling and temporal integration over multiple frames.

There has been considerable interest in distributed source coding (DSC) in recent years, primarily due to its potential contributions to low-power sensor networks. However, two major obstacles pose an existential threat to practical deployment of such techniques: the exponential growth of decoding complexity with network size and coding rates and the critical requirement of resilience to bit errors and erasures, given the severe channel conditions in many wireless sensor network applications. This article proposes a novel, unified approach for large-scale, error/erasure-resilient DSC that incorporates an optimally designed, nearest neighbor classifier-based decoding framework, where the design explicitly controls performance versus decoding complexity. Motivated by the highly nonconvex nature of the cost function, we present a deterministic annealing-based optimization algorithm for the joint design of the system parameters, which further enhances the performance over the greedy iterative descent technique. Simulation results on both synthetic and real sensor network data provide strong evidence for performance gains compared to other state-of-the-art techniques and may open the door to practical deployment of DSC in large sensor networks. Moreover, the framework provides a principled way to naturally scale to large networks while constraining decoder complexity, thereby enabling performance gains that increase with network size.

There has been considerable interest in distributed source coding within the compression and sensor network research communities in recent years, primarily due to its potential contributions to low-power sensor networks. However, two major obstacles pose an existential threat on practical deployment of such techniques in real world sensor networks, namely, the exponential growth of decoding complexity with network size and coding rates, and the critical requirement for error-resilience given the severe channel conditions in many wireless sensor networks. Motivated by these challenges, this paper proposes a novel, unified approach for large scale, error-resilient distributed source coding, based on an optimally designed classifier-based decoding framework, where the design explicitly controls the decoding complexity. We also present a deterministic annealing (DA) based global optimization algorithm for the design due to the highly non-convex nature of the cost function, which further enhances the performance over basic greedy iterative descent technique. Simulation results on data, both synthetic and from real sensor networks, provide strong evidence that the approach opens the door to practical deployment of distributed coding in large sensor networks. It not only yields substantial gains in terms of overall distortion, compared to other state-of-the-art techniques, but also demonstrates how its decoder naturally scales to large networks while constraining the complexity, thereby enabling performance gains that increase with network size.

Canonical distributed quantization schemes do not scale to large sensor networks due to the exponential decoder storage complexity that they entail. Prior efforts to tackle this issue have largely been limited to the suboptimal schemes of source grouping and decoding, thus failing to use all available information at the decoder. We propose a new decoding paradigm where all received bits are used in decoding. Essentially, to decode each source, we partition the space of received bit-tuples using a nearest neighbor quantizer at a decoding rate consistent with the allowed complexity and each partition is then mapped to a reconstruction value for that source. To avoid local minima in design, we resort to deterministic annealing to determine the nearest neighbor partition function (the partitioning prototypes) from the training set. Results on several data-sets show substantial gains over naive and other competing approaches.

We consider approaches for similarity search in correlated, high-dimensional data sets, which are derived within a clustering framework. We note that indexing by “vector approximation” (VA-File), which was proposed as a technique to combat the “Curse of Dimensionality,” employs scalar quantization, and hence necessarily ignores dependencies across dimensions, which represents a source of suboptimality. Clustering, on the other hand, exploits interdimensional correlations and is thus a more compact representation of the data set. However, existing methods to prune irrelevant clusters are based on bounding hyperspheres and/or bounding rectangles, whose lack of tightness compromises their efficiency in exact nearest neighbor search. We propose a new cluster-adaptive distance bound based on separating hyperplane boundaries of Voronoi clusters to complement our cluster based index. This bound enables efficient spatial filtering, with a relatively small preprocessing storage overhead and is applicable to euclidean and Mahalanobis similarity measures. Experiments in exact nearest-neighbor set retrieval, conducted on real data sets, show that our indexing method is scalable with data set size and data dimensionality and outperforms several recently proposed indexes. Relative to the VA-File, over a wide range of quantization resolutions, it is able to reduce random IO accesses, given (roughly) the same amount of sequential IO operations, by factors reaching 100X and more.

This paper considers the problem of distributed source coding for a large sensor network. A typical shortcoming of current approaches to true distributed coding is the exponential growth of the decoder codebook size with the number of sources in the network. This growth in complexity renders many traditional approaches impractical for even moderately sized sensor networks. Inspired by our recent results on fusion coding for selective retrieval, we propose a new distributed coding approach that scales to a large number of sources. Central to our approach is a “bit-subset selector” module whose role is to judiciously extract an appropriate subset of the received bits for decoding per individual source. This, together with joint design of all system components, enables direct optimization of the decoder complexity-distortion tradeoff, and thereby the desired scalability. Experiments on both real and synthetic data-sets show considerable gains over heuristic schemes.

Motivated by the need to efficiently leverage user relevance feedback in content-based retrieval from image databases, we propose a fast, clustering-based indexing technique for exact nearest-neighbor search that adapts to the Mahalanobis distance with a varying weight matrix. We derive a basic property of point-to-hyperplane Mahalanobis distance, which enables efficient recalculation of such distances as the Mahalanobis weight matrix is varied. This property is exploited to recalculate bounds on query-cluster distances via projection on known separating hyperplanes (available from the underlying clustering procedure), to effectively eliminate noncompetitive clusters from the search and to retrieve clusters in increasing order of (the appropriate) distance from the query. We compare performance with an existing variant of VA-File indexing designed for relevance feedback, and observe considerable gains.

Motivated by sensor networks, we consider the fusion storage of correlated sources in a database, such that any subset of them may be efficiently retrieved in the future. Only statistical information about future queries is available during encoding and storage. Fusion coding of correlated sources poses new challenges due to the conflicting objectives of exploiting inter-source correlations and enabling efficient selective retrieval. Practical signal compression imposes additional constraints on system complexity. We propose a shared-descriptions approach for the design of lossy fusion coding systems, to manage the precise tradeoffs between storage rate, retrieval rate, distortion and system complexity, within one unified framework. An iterative descent algorithm is derived for the design of such fusion coders. The optimized system provides significant gains over traditional quantization techniques that are not directly optimized for fusion coding.

This paper considers the problem of predictive fusion coding for storage of multiple spatio-temporally correlated sources so as to enable efficient selective retrieval of data from subsets of sources as designated by future queries. Only statistical information about future queries is available during encoding. While temporal correlations can be exploited by coding over large blocks, the growth in encoding complexity renders this approach impractical and hence the interest in a low complexity predictive coding approach. However, the design of optimal predictive fusion coding systems is considerably complicated by the presence of the prediction loop, and the potentially exponential growth of the query sets. We propose a complexity-constrained predictive fusion coder and derive an iterative algorithm for its design, which is based on the "Asymptotic Closed Loop" framework and hence, circumvents convergence and stability issues of traditional predictive quantizer design. The proposed predictive fusion coder optimizes the distortion - retrieval rate tradeoff, given a fixed storage capacity, and provides significant gains over storage schemes that perform only joint compression or memoryless fusion coding of all sources.

Motivated by the need to efficiently leverage user relevance feedback in content-based retrieval from image databases, we propose a fast, clustering-based indexing technique for exact nearest-neighbor search that adapts to the Mahalanobis distance with a varying weight matrix. We derive a basic property of point-to-hyperplane Mahalanobis distance, which enables efficient recalculation of such distances as the Mahalanobis weight matrix is varied. This property is exploited to recalculate bounds on query-cluster distances via projection on known separating hyperplanes (available from the underlying clustering procedure), to effectively eliminate noncompetitive clusters from the search and to retrieve clusters in increasing order of (the appropriate) distance from the query. We compare performance with an existing variant of VA-File indexing designed for relevance feedback, and observe considerable gains.

We consider the problem of storing data from multiple correlated sources in a database, so as to enable efficient future retrieval of data from select subsets of the sources, where only statistical information about queries may be available in advance. This setting poses new challenges in terms of the precise tradeoffs between storage rate, retrieval rate and distortion. We derive a gradient descent algorithm to optimize the joint encoding (storage) procedure, the decoder, and the retrieval procedure via mapping from queries to subsets of the stored data to retrieve and decode, so that the average retrieval rate-distortion cost is minimized, given a pre-specified overall storage capacity (or rate). Experiments conducted on real and synthetic data-sets demonstrate that our selective retrieval procedure is able to achieve significantly better trade-offs than joint compression, with retrieval speed-ups reaching 5X and distortion gains of up to 3.5 dB possible.

We consider approaches for exact similarity search in a high dimensional space of correlated features representing image datasets, based on principles of clustering and vector quantization. We develop an adaptive cluster distance bound based on separating hyperplanes, that complements our index in selectively retrieving clusters that contain data entries closest to the query. Experiments conducted on real data-sets confirm the efficiency of our approach with random disk IOs reduced by 100X, as compared with the popular vector approximation-file (VA-File) approach, when allowed (roughly) the same number of sequential disk accesses, with relatively low preprocessing storage and computational costs.

Motivated by the sensor network setting, we consider lossless storage of correlated discrete memoryless sources. The underlying tradeoff is between exploitation of inter-source correlation for low rate storage and efficient (low rate) selective retrieval from the fusion storage. We define the problem of shared descriptions (SD) source coding and relate it to the storage and retrieval problem. We present an achievable rate region for the SD problem and use it to characterize the storage vs. retrieval tradeoff