The proposed framework supports a new and efficient non-rigid body object tracking among objects with similar patterns. Human tracking is used as an example of a non-rigid body tracking. The main objective of this framework is to track the targeted person among people with similar patterns in a series of images frames. The targeted person is identified in the initial frame by the user. This framework consists of three stages: generation of panoramic images for a wider range, detection stage, and tracking stage. In detection stage, all multiple regions of interest (ROIs) are classified and the target is detected using multiple ROIs and fuzzy neural system. In tracking stage, the targeted person is tracked using an adaptive motion frame method that checks the target's movement. This suggested framework explains how multiple ROIs are used for detecting non-rigid body object and how the target can be tracked using previous trajectory and velocity. This algorithm contributes towards the tracking of a desired target that exists among many similar non-rigid body objects.