Differential privacy in federated learning has emerged as a promising solution for big data applications to achieve strong privacy guarantees. While prior work assumes that the privacy requirements are homogeneous across all clients, in practice, privacy requirements can differ across clients. In this paper, we introduce a cohort-based (ϵ, δ)-DP framework where privacy requirements and data distribution differ across these client cohorts. We show that the performance of existing differentially private stochastic algorithms degrade significantly for heterogeneous privacy scenarios, especially when the data is non-independent and identically distributed (non-iid). Moreover, we propose two novel continual learning-based DP training methods — DP-Synaptic intelligence (DP-SI) and DP-Rehearsal (DP-R) — to improve the model performance of cohorts with heterogeneous privacy budgets. We evaluate our approach on real datasets and show that our techniques outperform baseline techniques. Furthermore, our approach adapts to post-hoc privacy budget relaxations, providing greater flexibility in training models without significantly impacting performance.