We present a real-time bidirectional communication system that lets two people, separated by distance, experience a face-to-face conversation as if they were copresent. It is the first telepresence system that is demonstrably better than 2D videoconferencing, as measured using participant ratings (e.g., presence, attentiveness, reaction-gauging, engagement), meeting recall, and observed nonverbal behaviors (e.g., head nods, eyebrow movements). This milestone is reached by maximizing audiovisual fidelity and the sense of copresence in all design elements, including physical layout, lighting, face tracking, multi-view capture, microphone array, multi-stream compression, loudspeaker output, and lenticular display. Our system achieves key 3D audiovisual cues (stereopsis, motion parallax, and spatialized audio) and enables the full range of communication cues (eye contact, hand gestures, and body language), yet does not require special glasses or body-worn microphones/headphones. The system consists of a head-tracked autostereoscopic display, high-resolution 3D capture and rendering subsystems, and network transmission using compressed color and depth video streams. Other contributions include a novel image-based geometry fusion algorithm, free-space dereverberation, and talker localization.

This work is aimed at developing a design for the use of a microphone array with binaural hearing aids. The goal of such a hearing aid is to provide both the spatial-filtering benefits of the array and the natural benefits to sound localization and speech intelligibility that accrue from binaural listening. The present study examines two types of designs for fixed-processing systems: one in which independent arrays provide outputs to the two ears, and another in which the binaural outputs are derived from a single array. For the latter, various methods are used to merge array processing with binaural listening. In one approach, filters are designed to satisfy a frequency-dependent trade between directionality and binaural cue fidelity. In another, the microphone signals are filtered into low- and high-frequency components with the lowpass signals providing binaural cues and the highpass signal being the single output of the array processor. Acoustic and behavioral measurements were made in an anechoic chamber and in a moderately reverberant room to evaluate example systems. Theoretical performance was calculated for model arrays mounted on an idealized spherical head. Results show that both single- and dual-array systems provided target-intelligibility enhancements (2-4 dB improvements in speech reception threshold) relative to binaural cardioid microphones. In addition, the binaural-output systems provided cues that assist in sound localization, with resulting performance depending directly upon the cue fidelity. Finally, the sphere-based calculations accurately reflected the major features of the actual head-mounted array results, both in terms of directional sensitivity and output binaural cues.

For pt.I see ibid., vol.5, no.6, p.529-42, 1997. This work is aimed at developing a design for the use of a microphone array with binaural hearing aids. The goal of such a hearing aid is to provide both the spatial-filtering benefits of the array and the natural benefits to sound localization ability and speech intelligibility that result from binaural listening. The present study examines a design in which two ear-level omnidirectional microphones constitute the array. The merging of array processing with binaural listening is accomplished by dividing the frequency spectrum, devoting the lowpass part to binaural processing and the highpass part to adaptive array processing. Acoustic and behavioral measurements were made in an anechoic chamber and in a moderately reverberant room to assess the trade-off between sound localization and speech reception as the cutoff frequency was varied. A lowpass/highpass cutoff frequency of 500 Hz provided an improvement of 40 percentage points in sentence intelligibility over unaided listening for normal-hearing listeners, while still allowing adequate localization performance. Comparison of this binaural adaptive system to traditional amplification configurations with normal-hearing listeners showed improvements in speech reception in noise in a mildly reverberant room of approximately 3 dB over simple binaural amplification and 5 dB over monaural amplification.