Streaming music has dominated the digital music industry in recent years, which allows users to enjoy a huge music library online with a low subscription price. Terminal-side audio DRM (Digital Right Management) is very critical for streaming music industry, compromising of which will cause unrestricted listening, dumping and unauthorized secondary distribution. However, existing DRM protection schemes mainly focus on defeating software attacks but lack complete shielding against the physical memory disclosure attacks, which may even be launched by the owner of the terminal device. In this paper, we propose a terminal-side audio DRM solution called TrustSAMP to protect the copyrighted audio data against both software attacks and physical memory disclosure attacks. The basic idea is to process the audio data plaintext only in certain on-SoC components secured by ARM TrustZone. To minimize the TCB (Trusted Computing Base) of the secure world, we separate the control flow and the data flow of the Linux audio subsystem and port only the codes used for audio data decryption and plaintext transfer into the secure world. Moreover, we leave most driver codes of the audio-associated on-SoC components in the rich OS (i.e., in the normal world), and introduce a tiny proxy in the secure world to control the associated registers according to the requests from the normal-world drivers. The prototype implemented on real hardware shows that TrustSAMP can play a variety of wav-format audio with very small overhead and negligible loss of audio quality.