Support vector machine (SVM) is an excellent pattern recognition method. Many experiments have shown that SVM can achieve a generalization performance gain by carrying out it in the feature transformation space. Nevertheless, the theoretical foundation behind this phenomenon is presently lack of deep investigation. In the paper, we first give and prove a vital theoretical conclusion that SVM in the feature transformation space can obtain a lower radius-margin bound than one in the feature original space. This means that the performance of SVM can be improved by feature transformation since the radius-margin bound is directly associated with the generalization capacity. Based on this theoretical support, we further propose a novel method called covering-hyperellipsoid-constrained large margin classifier (CHC-LMC). The key characteristic of CHC-LMC is that it jointly learns the minimum bounding hyperellipse and the used classifier by directly minimizing the radius-margin bound in the transformation space, and so embodies the structural risk minimization principle. We develop the linear and nonlinear versions of CHC-LMC and employ an alternate optimization strategy to deal with the corresponding optimization problems. Finally, comprehensive experiments are conducted to verify the validity of CHC-LMC and evaluate the generalization performance by comparing it with the competing methods.