Traditionally, automatic speech recognition (ASR) systems rely on human transcriptions to calculate word error rate (WER) by comparing ASR outputs to manual transcriptions. Recently, Amazon's mobile voice shopping platform stopped storing audio from incoming requests to enhance customer privacy, making offline, human-based evaluation unfeasible. This presentation introduces a multitask Speech LLM-based system that processes real-time audio, extracting key features to track ASR performance and detect traffic shifts-all without storing audio or requiring human annotations. Additionally, we demonstrate how combining these features with a synthetic audio generation model (TTS) enables accurate detection of ASR performance degradation, ensuring continuous optimization of the customer voice experience.