We are witnessing an emergence in Passive Sensing enabled AI (PSAI) to provide dynamic insights for performance and wellbeing of information workers. Hybrid work paradigms have simultaneously created new opportunities for PSAI, but have also fostered anxieties of misuse and privacy intrusions within a power asymmetry. At this juncture, it is unclear if those who are sensed can find these systems acceptable. We conducted scenario-based interviews of 28 information workers to highlight their perspectives as data subjects in PSAI. We unpack their expectations using the Contextual Integrity framework of privacy and information gathering. Participants described appropriateness of PSAI based on its impact on job consequences, work-life boundaries, and preservation of flexibility. They perceived that PSAI inferences could be shared with selected stakeholders if they could negotiate the algorithmic inferences. Our findings help envision worker-centric approaches to implementing PSAI as an empowering tool in the future of work.