We combine the often opposing forces of artistic freedom and mathematical determinism to enrich a given animation or simulation of a surface with physically based detail. We present a process called tracking , which takes as input a rough animation or simulation and enhances it with physically simulated detail. Building on the foundation of constrained Lagrangian mechanics, we propose weak-form constraints for tracking the input motion. This method allows the artist to choose where to add details such as characteristic wrinkles and folds of various thin shell materials and dynamical effects of physical forces. We demonstrate multiple applications ranging from enhancing an artist's animated character to guiding a simulated inanimate object.