Capturing fine-grained 3D shapes and anthropometric features of human ears is a non-trivial and yet important task in a variety of applications. Existing approaches relying on specialized equipment such as high-definition 3D scanners or photogrammetry from RGB(-D) cameras are laborious in data acquisition and post-processing, and may perform poorly in case of occlusions from face and head covering, accessories, and the complex geometry of pinnae. Recognizing the strong dependency of acoustic anatomical transfer functions called Head-Related Transfer Functions (HRTFs) on the morphological features of pinnae, we develop HRTF2Mesh, a novel acoustic-based method for 3D human ear mesh reconstruction using commodity devices – a pair of earphone microphones and a smartphone device. HRTF2Mesh takes sparse acoustic measurement data from a subject, estimates and encodes the HRTFs as low-dimension vectors and then generates 3D mesh representation of their ear with the help of 3D morphable ear models. HRTF2Mesh has been evaluated using both public datasets and data collected from a diverse set of volunteers. Experiment results show that HRTF2Mesh achieves an average ear mesh estimation errors of 0.92 mm for test subjects in public datasets, and 1.70 mm amongst the volunteers. The data acquisition process of HRTF2Mesh can be completed under 5 minutes by a single user.

The prevalence of earable devices, such as earbuds and headphones, allows people to converse or listen to audio recordings on the move but often at the cost of reduced alertness of imminent health and safety threats in their surroundings. 3-D binaural sound localization (BSL), which aims to locate sound sources in space, plays a key role in improving one’s situation awareness. BSL on earable devices is inherently challenging due to the limited number of microphones available as well as subject- and location-dependent filtration effects of a person’s pinna, head and torso, described by head-related transfer functions (HRTFs). In this work, we develop DeepBSL, a deep neural network model to estimate azimuth and elevation angles of sound sources relative to a person’s head. For a new subject, an efficient procedure is developed to collect HRTFs at sparse locations using in-ear microphones and a mobile phone, which are then utilized to synthesize sounds of any type at arbitrary locations to train personalized DeepBSL models. Extensive evaluations using synthetic data from a public data set and through real-world experiments demonstrate that the personalized DeepBSL models are data-efficient and can achieve better-than-human performances in BSL while significantly outperforming a state-of-the-art model that can only predict azimuth angles of sound sources. Our best performing model has an average azimuth prediction error of$2.9^{\circ } (4.1^{\circ })$and elevation prediction error of$1.4^{\circ } (2.9^{\circ })$in an indoor (outdoor) environment.

A Head Related Transfer Function (HRTF) characterizes how a hu-man ear receives sounds from a point in space, and depends on the shapes of one's head, pinna, and torso. Accurate estimations of HRTFs for human subjects are crucial in enabling binaural acoustic applications such as sound localization and 3D sound spatialization. Unfortunately, conventional approaches for HRTF estimation rely on specialized devices or lengthy measurement processes. This work proposes a novel lightweight method for HRTF individual-ization that can be implemented using commercial-off-the-shelf components and performed by average users in home settings. The proposed method has two key components: a generative neural network model that can be individualized to predict HRTFs of new subjects from sparse measurements, and a lightweight measurement procedure that collects HRTF data from spatial locations. Exten-sive experiments using a public dataset and in house measurement data from 10 subjects of different ages and genders, show that the individualized models significantly outperform a baseline model in the accuracy of predicted HRTFs. To further demonstrate the advantages of individualized HRTFs, we implement two prototype applications for binaural localization and acoustic spatialization. We find that the performance of a localization model is improved by 15° after trained with individualized HRTFs. Furthermore, in hearing tests, the success rate of correctly identifying the azimuth direction of incoming sounds increases by 183% after individualization.