We present an approach that uses detailed 3D models to detect and classify objects into fine levels of vehicle categories. Unlike other approaches that use silhouette information to fit a 3D model, our approach uses complete appearance from the image. Each 3D model has a set of salient location markers that are determined a-priori. These salient locations represent a sub-sampling of 3D locations that make up the model. Scene conditions are simulated in the rendering of 3D models and the salient locations are used to bootstrap a HoG based feature classifier. HoG features are computed in both rendered and real scenes and a novel object match score the `Salient Feature Match Distribution Matrix' is computed. For each 3D model we also learn the patterns of misalignment with other vehicle types and use it as an additional cue for classification. Results are presented on a challenging aerial video dataset consisting of vehicle imagery from various viewpoints and environmental conditions.