With the desired functionality of moving object tracking, wireless pan-tilt cameras are able to play critical roles in a growing diversity of surveillance environments. However, today's pan-tilt cameras oftentimes underperform when tracking frequently moving objects like humans – they are prone to lose sight of objects and bring about excessive mechanical rotations that are especially detrimental to those energy-constrained outdoor scenarios. The ineffectiveness and high cost of all state-of-the-art tracking approaches are rooted in their adherence to the industry's simplicity principle, which leads to their stateless nature, performing gimbal rotations based only on the latest object detection. To address the issues, we design and implement WiseCam that wisely tunes the pan-tilt cameras to minimize mechanical rotation costs while maintaining long-term object tracking. This systematic tracking approach also tackles issues of motion-rotation speed gap and scattered moving objects, which is universally applicable to complex tracking scenarios. We examine the performance of WiseCam by experiments on two types of pan-tilt cameras with different motors. Results show that it significantly outperforms the state-of-the-art tracking approaches on both tracking duration and power consumption.