Background. Autonomous systems, whether AI-enabled or not, are ubiquitous and pervasive in our daily lives. While their adoption and use bring many benefits, they also pose significant ethical challenges. Objective. The objective of this work is to contribute a reference architecture for ethical-aware autonomous systems, focusing on their interaction and collaboration with humans, being them proactive, reactive or passive in the interaction with the systems. Method. To define the architecture, we analyzed scientific papers in the field, guidelines and recommendations, as well as laws and regulations. We then applied this acquired knowledge to build the reference architecture. The results of this work were validated through expert interviews and a scenario-based evaluation. Results. We contribute (i) a definition of ethical-aware autonomous systems, (ii) requirements for ethical-aware autonomous systems, and (iii) a reference architecture for ethical-aware autonomous systems. Our reference architecture is intended to help system and software engineers to design autonomous or intelligent systems that should interact and operate with humans in ethically sensitive contexts such as healthcare, social robotics, and assistive technologies. Conclusion. We believe that this work will assist software architects and engineers in designing and developing autonomous systems that should interact and collaborate with humans while respecting values important to individuals, society, and the environment.

The presence of autonomous systems is growing at a fast pace and it is impacting many aspects of our lives. Designed to learn and act independently, these systems operate and perform decision-making without human intervention. However, they lack the ability to incorporate users’ ethical preferences, which are unique for each individual in society and are required to personalize the decision-making processes. This reduces user trust and prevents autonomous systems from behaving according to the moral beliefs of their end-users. When multiple systems interact with differing ethical preferences, they must negotiate to reach an agreement that satisfies the ethical beliefs of all the parties involved and adjust their behavior consequently. To address this challenge, this paper proposes RobEthiChor , an approach that enables autonomous systems to incorporate user ethical preferences and contextual factors into their decision-making through ethics-based negotiation. RobEthiChor features a domain-agnostic reference architecture for designing autonomous systems capable of ethic-based negotiating. The paper also presents RobEthiChor-Ros , an implementation of RobEthiChor within the Robot Operating System (ROS), which can be deployed on robots to provide them with ethics-based negotiation capabilities. To evaluate our approach, we deployed RobEthiChor-Ros on real robots and ran scenarios where a pair of robots negotiate upon resource contention. Experimental results demonstrate the feasibility and effectiveness of the system in realizing ethics-based negotiation. RobEthiChor allowed robots to reach an agreement in more than 73 % of the scenarios with an acceptable negotiation time (0.67s on average). Experiments also demonstrate that the negotiation approach implemented in RobEthiChor is scalable.

Large Language Models (LLMs) are increasingly integrated into software engineering (SE) tools for tasks that extend beyond code synthesis, including judgment under uncertainty and reasoning in ethically significant contexts. We present a fully automated framework for assessing ethical reasoning capabilities across 16 LLMs in a zero-shot setting, using 30 real-world ethically charged scenarios. Each model is prompted to identify the most applicable ethical theory to an action, assess its moral acceptability, and explain the reasoning behind their choice. Responses are compared against expert ethicists’ choices using inter-model agreement metrics. Our results show that LLMs achieve an average Theory Consistency Rate (TCR) of 73.3% and Binary Agreement Rate (BAR) on moral acceptability of 86.7%, with interpretable divergences concentrated in ethically ambiguous cases. A qualitative analysis of free-text explanations reveals strong conceptual convergence across models despite surface-level lexical diversity. These findings support the potential viability of LLMs as ethical inference engines within SE pipelines, enabling scalable, auditable, and adaptive integration of user-aligned ethical reasoning. Our focus is the Ethical Interpreter component of a broader profiling pipeline: we evaluate whether current LLMs exhibit sufficient interpretive stability and theory-consistent reasoning to support automated profiling.

Abstract Autonomous intelligent systems are known as artificial intelligence software entities that can act on their own and can take decisions without any human intervention. The communication between such systems to reach an agreement for problem-solving is known as automated negotiation. This study aims to systematically identify and analyze the literature on automated negotiation from four distinct viewpoints: (1) the existing literature on negotiation with focus on automation, (2) the specific purpose and application domain of the studies published in the domain of automated negotiation, (3) the input, and techniques used to model the negotiation process, and (4) the limitations of the state of the art and future research directions. For this purpose, we performed a systematic mapping study (SMS) starting from 73,760 potentially relevant studies belonging to 24 conference proceedings and 22 journal issues. Through a precise selection procedure, we identified 50 primary studies, published from the year 2000 onward, which were analyzed by applying a classification framework. As a result, we provide: (a) a classification framework to analyze the automated negotiation literature according to several parameters (e.g., focus of the paper, inputs required to carry on the negotiation process, techniques applied, and type of agents involved in the negotiation), (b) an up-to-date map of the literature specifying the purpose and application domain of each study, (c) a list of techniques used to automate the negotiation process and the list of input to carry out the negotiation, and (d) a discussion about promising challenges and their consequences for future research. We also provide a replication package to help researchers replicate and verify our systematic mapping study. The results and findings will benefit researchers and practitioners in identifying the research gap and conducting further research to bring dedicated solutions for automated negotiation.

This paper briefly outlines a high-level architecture of a context-aware ethics-based negotiation approach in which autonomous systems utilize user ethical profiles, together with contextual factors and user status, to control their autonomy while collaboratively negotiating to reach an ethical agreement that satisfies the ethical beliefs of all parties involved.