Specification learning and controller synthesis are two methods that promise to provide control systems with assured adaptive capabilities at run-time. Specification learning can automatically update specifications in light of violation traces observed within the operational environment. Controller synthesis can then automatically generate implementations that are guaranteed to satisfy these specifications in every environment.Specification learning is implemented using general-purpose AI systems. These systems are highly configurable, and the configuration choice heavily affects the effectiveness. Setting configuration parameters is far from obvious as they bear no clear semantic relation with the adaptation task. State of the art requires configurations to be set by domain experts at design time for each application domain.In this paper, we argue that to create assured control systems that can effectively and efficiently adapt at run-time, the learning systems upon which they are built must also have adaptive learning strategies for determining configurations at runtime. We demonstrate this idea with a proof-of-concept that computes domain-dependent policies using reinforcement learning.