Due to their limited ability to reason about the social context in which they are used, smart speakers pose significant privacy risks by responding in ways that may violate people's implicit social boundaries. We conducted a cross-cultural vignette study (N = 944) in Germany and Singapore to investigate how situational factors—specifically social context (bystander relationships and closeness), physical context (location), and interaction context (topic and deceptive intent)—regulate user preferences for smart speaker responses. Our results demonstrate that these factors are superior predictors of response preferences than dispositional user traits (i.e., intrinsic personal traits). We identify two distinct social dynamics: a structural influence for professional relationships (e.g., boss) that persists regardless of social closeness, and a closeness-based influence for personal relationships. We further demonstrate that deceptive intent drives privacy-seeking behaviour, acting as a tool for social impression management. In parallel, we evaluate how LLMs respond to the same scenarios, revealing mismatches between model behaviour and human expectations, particularly in culturally contingent situations. These mismatches expose new privacy risks arising from socially miscalibrated AI reasoning. We conclude with design strategies to better align device behaviour with these social nuances.