Diabetic retinopathy (DR) is a complication of diabetic Mellitus, developing retinal lesions that impair vision. The DR detection in the early stages avoids permanent vision loss. The treatments provide relief, and the vision loss due to DR is irreversible. The manual grading of DR is time-consuming and prone to human errors. The other real-time problem is exchanging patient fundus image information with hospitals worldwide while upholding the organisations’ privacy concerns. When training a deep learning (DL) network, two critical factors to keep in mind are creating a collaborative platform and protecting patient data privacy. Therefore, an automated DR detection technique is required while protecting patient data and privacy. In this work, we propose a novel DR severity grading technique based on Federated Learning (FL), a recent advancement in DL called DRFL. FL is a new research paradigm that allows DL models to be trained collectively without disclosing clinical information. In DRFL, we combined the Federated averaging (FedAvg) technique and the median of the categorical cross-entropy loss. Since in comparison to FedAvg, the median cross-entropy is better suited for either under-fitted or over-fitted clients. Also, we propose a novel central server that extracts multi-scale features from the fundus images to identify small lesions present in the fundus image. In this work, we consider five clients holding different preprocessed fundus images collected from publicly available databases such as MESSIDOR-2, IDRiD, Kaggle, and a local database collected from Silchar Medical College and Hospital. The proposed model obtained an accuracy of 98.6%, specificity of 99.3%, precision of 97.25%, and an F1 score of 97.5%, which are better results than other techniques.