We study the relation between the structure of algebraic and context-free subsets of a group G and that of a finite index subgroup H. Using these results, we prove that a kind of Fatou property, previously studied by Berstel and Sakarovitch in the context of rational subsets and by Herbst in the context of algebraic subsets, holds for context-free subsets if and only if the group is virtually free. We also exhibit a counterexample to a question of Herbst concerning this property for algebraic subsets.