Currently, the entire world is fighting against the Corona Virus (COVID-19). As of now, more than thirty lacs of people all over the world were died due to the COVID-19 till April 2021. A recent study conducted by China suggests that Chest CT and X-ray images can be used as a preliminary test for COVID detection. This paper propose a transfer learning-based mathematical COVID detection model, which integrates a pre-trained model with the Random Forest Tree (RFT) classifier. As the available COVID dataset is noisy and imbalanced so Principal Component Analysis (PCA) and Generative Adversarial Networks (GANs) is used to extract most prominent features and balance the dataset respectively. The Bayesian Cross-Entropy Loss function is used to penalize the false detection differently according to the class sensitivity (i.e., COVID patient should not be classified as Normal or Pneumonia class). Due to the small dataset, a pre-trained model like VGGNet-19, ResNet50 and Inception_ResNet_V2 were chosen to extract features and then trained them over the RFT for the classification task. The experiment results showed that ResNet50 gives the maximum accuracy of 99.51%, 98.21%, and 97.2% for training, validation, and testing phases, respectively, and none of the COVID Chest X-ray images were classified as Normal or Pneumonia classes.