To fulfill the needs for accurate trajectory tracking with large displacement in a handheld instrument, a 3-DOF serial compliant mechanism is developed. The mechanism is compact with a total length less than 150 mm and a maximum diameter of 22 mm. Two flexures are developed using different rapid prototyping techniques: one 3-DOF flexural lever made of Vero-Gray by Polyjet and a 1-DOF translational flexure made of stainless steel by Direct Metal Laser Sintering (DMLS). Analytical and Finite Element (FE) models are developed for the proposed flexural mechanisms. Experiments are conducted on a prototype. To improve the tracking accuracy, the hysteretic nonlinearities of the system are modeled using Prandtl-Ishlinskii model. Inverse feedforward controller is implemented to linearize the relationship between input and output. The tracking errors are reduced while maintaining a fast response of the system. The total tracking errors are identified individually for each axis and then compensated. Tracking performances of the tool tip are evaluated experimentally with different inputs. The RMS tracking error of the proposed mechanism is lower than 1 µm in all axes, which is improved more than four times compared to the previous systems.