Popular consumer Internet of Things (IoT) devices provide increasingly diverse sensing and actuation capabilities. Despite their benefits, such devices prompt numerous security concerns. Typically, security is attained at device-level granularity, which relies upon device trustworthiness. However, if a device is compromised (e.g., via remote exploits), this approach fails. To this end, we construct SA4P: Sensing and Actuation as a Privilege, a framework that decouples IoT devices from their physical environment. In SA4P, whenever any software on a device wants to access a sensing or actuation peripheral, it must be authorized to do so. This is achieved by the inclusion of an on-board component, Peripheral Guard (PEG), that physically guards peripherals. Besides providing strong security guarantees, SA4P motivates developers to consider sensing and actuation as valuable resources. SA4P' design is modular, lightweight, and formally verified. It also does not require any hardware modifications for trusted execution environment (TEE)-equipped devices, while imposing only modest changes for other devices.