Conventional machine learning involves training models on centralized storage locations such as servers or databases, which pose security and privacy threats. Federated learning (FL) overcomes such limitations through collaborative global model training instead of centralized processing without sharing the private data in raw format, ensuring end-to-end integrity and confidentiality and protecting clients’ privacy. However, security issues such as local data leakage through shared gradients are a potential threat in FL. An attacker may forge sensitive information from the client’s local model updates and corrupt the aggregated global model result without detection, resulting in clients’ privacy vulnerability. This paper introduces SeFL, a privacy-preserving FL framework empowered by aggregator-oblivious (AO) techniques to ensure secure model aggregation between clients and a server. The aggregation server protects the masked gradients without revealing local data privacy. SeFL introduces a dynamic client management system which is robust against client dropout. Security analysis shows that SeFL can prevent the privacy-preserving requirements in FL. A thorough experiment is performed with various deep learning architectures using diverse datasets to evaluate the proposed framework. Results show that SeFL achieves comparable training accuracy with FedAvg and low computation & communication overhead when implementing the proposed privacy-preserving scheme.