Given the potent and pervasive nature of modern technologies, this paper lays out the complexities involved in achieving responsible design. In order to do this we will first compare an emerging policy-oriented programme of research known as RRI (Responsible Research and Innovation) with initiatives in HCI. A focus on the similarities and differences may highlight to what extent responsibility is already and successfully embedded within the concerns and practices of design and use, and what may yet need to be incorporated for responsible design. The paper then discusses the challenges of 'naturalising' the very ambitious programme of RRI within specific design activities and concerns, through the lens of four analytic concepts: reflexivity; responsiveness; inclusion; and anticipation. Finally, we make a case for a pragmatic, 'unromantic', but engaged reinterpretation of RRI for HCI.