In this paper, a robust, real-time object tracking approach capable of dealing with multiple symmetric and non-symmetric objects in a real-time requirement setting is proposed. The approach relies only on depth data to track multiple objects in a dynamic environment and uses random-forest based learning to deal with problems like object occlusion, motion-blur due to camera motion and clutter. We show that the relation between object motion and the corresponding change in its 3D point cloud data can be learned using only 6 random forests. A framework that unites object pose estimation and object pose tracking to efficiently track multiple objects in 3D space is presented. The approach is robust in tracking objects even in presence of motion blur that causes noisy depth data and is capable of real-time performance with 1.8ms per frame. The experimental evaluations demonstrate the performance of the approach against robustness, accuracy and speed and compare the approach with the state of the art. A publicly available dataset with real-world data is also provided for future benchmarking.