Cameras are widely adopted as the sensing device in fiducial marker systems. The captured videos however may expose sensitive information to attackers. We propose a privacy-preserving fiducial marker system, ReMark, which is based on retroreflector and single-pixel imaging (SPI), and captures only the minimal information needed for positioning and identifying markers. ReMark is built upon a state-of-the-art light-weight neural network (NN), which recovers the scene in the SPI system. Another light-weight NN is proposed for pose estimation of markers. They are trained with synthesized large datasets, and proved to adapt well to real-world data. The pose estimation NN adopts a specialized output embedding to address the symmetry-related issues, and soft-decision decoding is used to mitigate the distortion in recovered scenes. Detailed evaluation shows that, with 4.8 cm markers at 3.00 m distance and 1.0 W LED power, ReMark achieves a decode error rate of 2.1% with tilt angle ≤ 30°, searching in a dictionary of size 1,000. Evaluation of worst-case scenarios shows that an attacker could hardly acquire sensitive information even with access to raw data.