Soft fingertips can easily achieve high friction contacts but their frictional moment around the contact normals makes manipulation control difficult. We propose a control scheme to manipulate an object by a multifingered hand with soft fingertips. The proposed scheme cancels the frictional moment according to the direction of the relative rotation between the fingertip and the object. Whatever the fingertip contact type is, accurate pose measurement is very important for the object manipulation. We also propose a systematic method to estimate the object pose with a complementary use of camera images and finger joint information. The proposed control scheme and the pose estimation method were combined and implemented in an experimental system. Results of the experiment show the effectiveness of the proposed pose estimation method and the overall control system for manipulating an object by a soft-fingered hand.