The method wherein a human expert diagnose a patient for COVID-19 with the help of a chest CT scan or X-ray image could be one of the most reliable methods. However, this method of diagnosis is challenging and non-scalable while considering limited medical-care infrastructure and disease spread rate. We train COVID-19 diagnosis models for classification using both the image modalities, chest CT scan and X-ray datasets. We have used fusion approach for multimodal data fusion and proposed two variants. The first model is trained using an automated deep learning approach and in the second model features from the images are extracted using transfer learning approach followed by fine tuning of model. The performance of these models are evaluated with metrics like testing accuracy, recall, precision and f1-score. False negatives are critical and to ensure a smaller number of false negatives, cost-sensitive learning is enforced. The cost-sensitive convnet model achieves an accuracy of 97%.