Abstract In practical use, language models (LM) must efficiently adapt to new tasks and knowledge while avoiding catastrophic forgetting, a requirement that sparked research on continual learning (CL). Despite advances, current CL methods still lack a unified solution that delivers strong knowledge transfer and parameter-efficient capacity management while preventing catastrophic forgetting, which limits effective use of task synergies under tight training and memory budgets. We bridge this gap by introducing Gated Expandable Parameter-Efficient Fine-Tuning (GE-PEFT), a novel approach that shares knowledge of previous tasks through leveraging a single, dynamically expanding PEFT module within LMs while selectively gating irrelevant previous tasks. Our experiments across multiple task-incremental CL benchmarks show that GE-PEFT outperforms existing state-of-the-art CL approaches in both full CL and few-shot settings. Our ablation and parameter sensitivity studies highlight the benefit of each proposed component, demonstrating that GE-PEFT offers a more efficient and adaptive solution for CL in LMs.