Robust 3D pose tracking of an object is a critical technique for various mobile sensing applications. Computer vision-based pose tracking method provides a cost-effective solution, but it is sensitive to occlusion and illumination change issues. In this work, we propose a novel visual-inertial sensor fusion framework and demonstrate the real-time implementation of a tightly-coupled sensor fusion algorithm: inertial perspective-n-point (IPNP) algorithm. With measurements from an inertial measurement unit (IMU), the prototype system only needs to detect two keypoints to track all six degrees of freedom of a planar object, e.g., a mobile X-ray detector, a 50% reduction on required number of keypoints, compared with the vision-based perspective-n-point algorithm.