The rising popularity of camera glasses challenges societal norms of recording bystanders and thus requires efforts to mediate privacy preferences. We present the first study on the wearers’ perspectives and explore privacy challenges associated with wearing camera glasses when bystanders are present. We conducted a micro-longitudinal diary study (N = 15) followed by exit interviews with existing users and people without prior experience. Our results show that wearers consider the currently available privacy indicators ineffective. They believe the looks and interaction design of the glasses conceal the technology from unaware people. Due to the lack of effective privacy-mediating measures, wearers feel emotionally burdened with preserving bystanders’ privacy. We furthermore elicit how this sentiment impacts their usage of camera glasses and highlight the need for technical and non-technical solutions. Finally, we compare the wearers’ and bystanders’ perspectives and discuss the design space of a future privacy-preserving ecosystem for wearable cameras.