With recent advances in artificial intelligence and the rapidly increasing importance of autonomous intelligent systems in society, it is becoming clear that artificial agents will have to be designed to comply with complex ethical standards. As we work to develop moral machines, we also push the boundaries of existing legal categories. The most pressing question is what kind of ethical decision-making our machines are actually able to engage in. Both in law and in ethics, the concept of agency forms a basis for further legal and ethical categorisations, pertaining to decision-making ability. Hence, without a cross-disciplinary understanding of what we mean by ethical agency in machines, the question of responsibility and liability cannot be clearly addressed. Here we make first steps towards a comprehensive definition, by suggesting ways to distinguish between implicit and explicit forms of ethical agency.