We present the hardware design and software pipeline for an ultra-low power device, in the form factor of a wearable badge, that supports energy efficient sensing, processing and wireless transfer of human voice commands and interactions. The proposed system, called PA2BLO, is envisioned to support both: (a) real-time, scalable, authorized voice based interaction and control of devices and appliances, and (b) longitudinal, low-power logging of natural voice interactions. PA2BLO in-troduces two key novel capabilities. First, it includes a low power, low-complexity voice authentication module that is able to reliably authenticate an authorized user only using low sampling rate (500 Hz) audio data. Second, to reduce concerns around inadvertent leakage of voice biometrics to less secure voice-driven services, PA2BLO uses a power-efficient, randomized pitch shifting technique that dramatically lowers the ability to perform speaker recognition while preserving instruction/speech comprehensibility. We describe PA2BLO's Cortex M4F-based micro-controller based hardware implementation, which is care-fully designed to eliminate redundant processing and consumes less than 50J of energy per hour of active voice capture and processing. Through both controlled and naturalistic studies, we show that the PA2BLO prototype is capable of authenticating user voice segments reliably (accuracy> 89.8%) and can operate for well over a day (using a supercapacitor charged within just one minute) while capturing 2+ hours of active speaker data.