In this paper, we propose a 3D finger detecting and tracking algorithm (3dFDT) based on a sliding window for RGB-D sequences. Microsoft Kinect is utilized as a 3D depth camera, which provides RGB images and depth data simultaneously. However the depth data have large noises when the background is white. We are focusing on tracking the up or down movements of the fingertips in order to apply in some applications such as playing the piano on the table or " in the air". First, the skin statistical ellipse model is applied to detect the hand skin region, and the pseudo-contours are removed to get the interested hand region. Then, the region center is considered as the center of the palm. Next, the fingertips are located precisely using the convex defect detection based on the refined hand contours. Furthermore, the depth information is aligned to the fingertips in the RGB channels. Finally, the sliding window strategy is employed to stabilize the depth information due to large noises from the depth space especially when dealing with the white surface as backgrounds. The experimental results show that our proposed method is effective, and it can be applied in the real-time applications for non-contact interactions.