The emergence of foundation models (FMs) has enabled the development of highly capable and autonomous agents, unlocking new application opportunities across a wide range of domains. Evaluating the architecture of agents is particularly important as the architectural decisions significantly impact the quality attributes of agents given their unique characteristics, including compound architecture, autonomous and non-deterministic behaviour, and continuous evolution. However, these traditional methods fall short in addressing the evaluation needs of agent architecture due to the unique characteristics of these agents. Therefore, in this paper, we present AgentArcEval, a novel agent architecture evaluation method designed specially to address the complexities of FM-based agent architecture and its evaluation. Moreover, we present a catalogue of agent-specific general scenarios, which serves as a guide for generating concrete scenarios to design and evaluate the agent architecture. We demonstrate the usefulness of AgentArcEval and the catalogue through a case study on the architecture evaluation of a real-world tax copilot, named Luna.

Foundation model-enabled generative artificial intelligence facilitates the development and implementation of agents, which can leverage distinguished reasoning and language processing capabilities to takes a proactive, autonomous role to pursue users’ goals. Nevertheless, there is a lack of systematic knowledge to guide practitioners in designing the agents considering challenges of goal-seeking (including generating instrumental goals and plans), such as hallucinations inherent in foundation models, explainability of reasoning process, complex accountability, etc. To address this issue, we have performed a systematic literature review to understand the state-of-the-art foundation model-based agents and the broader ecosystem. In this paper, we present a pattern catalogue consisting of 18 architectural patterns with analyses of the context, forces, and trade-offs as the outcomes from the previous literature review. We propose a decision model for selecting the patterns. The proposed catalogue can provide holistic guidance for the effective use of patterns, and support the architecture design of foundation model-based agents by facilitating goal-seeking and plan generation. • A collection of architectural patterns for real-world agent implementations. • FM-based agent ecosystem with architectural pattern annotations as a guidance. • Curated analysis of patterns including benefits, trade-offs, and real-world uses. • A decision model for structuring the patterns and making rational design decisions.

Background:Along with the rapid development and widespread adoption of blockchain technology, many common practices have been summarized into blockchain-based design patterns for application development. However, the numerous and scattered patterns may cause confusion among practitioners. Therefore, adopting appropriate patterns to meet various requirements has become a major challenge, as it requires deep development experience and blockchain technology knowledge.Objective:To address this problem, this paper proposes a decision-support solution to assist with the selection of design patterns during the blockchain-based application development, including a layered taxonomy of design patterns, mappings of quality attributes with the patterns, and a decision model incorporating the taxonomy and mappings.Method:We collected 72 distinct and state-of-the-art design patterns via a Systematic Literature Review (SLR) to establish a layered taxonomy, and 18 unified quality attribute metrics were proposed for blockchain-based pattern assessment and mapping establishment. Based on the pattern taxonomy and quality attribute mappings, we developed a decision model that can provide intuitive guidance for pattern selection.Results:The proposed solution was evaluated through a case study in a seafood supply chain, in which we examined how well the decision model could help identify design flaws and provide reasonable solutions. Additionally, interviews and a questionnaire-based survey were conducted to measure the completeness, correctness, and usefulness of the proposed decision model. The evaluation results indicate that the proposed decision-support solution provides developers with comprehensive guidance, facilitates targeted decision making, and supports intuitive understanding.Conclusions:Our decision-support solution can improve the development efficiency of blockchain-based applications, especially in addressing potential design flaws, achieving targeted quality attributes, and reducing development costs.

Large language model (LLM) based chatbots, such as ChatGPT, have attracted huge interest in foundation models. It is widely believed that foundation models will serve as the fundamental building blocks for future AI systems. However, the architecture design of foundation model based systems has not yet been systematically explored. There is limited understanding about the impact of introducing foundation models in software architecture. Therefore, in this paper, we propose a taxonomy of foundation model based systems, which classifies and compares the characteristics of foundation models and system design options. Our taxonomy comprises three categories: the pretraining and adaptation of foundation models, the architecture design of foundation model based systems, and responsible-AI-by-design. This taxonomy can serve as concrete guidance for designing foundation model based systems.

Artificial Intelligence (AI), particularly through the advent of large-scale generative AI (GenAI) models such as Large Language Models (LLMs), has become a transformative element in contemporary technology. While these models have unlocked new possibilities, they simultaneously present significant challenges, such as concerns over data privacy and the propensity to generate misleading or fabricated content. Current frameworks for Responsible AI (RAI) often fall short in providing the granular guidance necessary for tangible application, especially for Accountability---a principle that is pivotal for ensuring transparent and auditable decision-making, bolstering public trust, and meeting increasing regulatory expectations. This study bridges the Accountability gap by introducing our effort towards a comprehensive metrics catalogue, formulated through a systematic multivocal literature review (MLR) that integrates findings from both academic and grey literature. Our catalogue delineates process metrics that underpin procedural integrity, resource metrics that provide necessary tools and frameworks, and product metrics that reflect the outputs of AI systems. This tripartite framework is designed to operationalize Accountability in AI, with a special emphasis on addressing the intricacies of GenAI.

The advent of Generative AI has marked a significant milestone in artificial intelligence, demonstrating remarkable capabilities in generating realistic images, texts, and data patterns. However, these advancements come with heightened concerns over data privacy and copyright infringement, primarily due to the reliance on vast datasets for model training. Traditional approaches like differential privacy, machine unlearning, and data poisoning only offer fragmented solutions to these complex issues. Our paper delves into the multifaceted challenges of privacy and copyright protection within the data lifecycle. We advocate for integrated approaches that combines technical innovation with ethical foresight, holistically addressing these concerns by investigating and devising solutions that are informed by the lifecycle perspective. This work aims to catalyze a broader discussion and inspire concerted efforts towards data privacy and copyright integrity in Generative AI.

The rapid development of artificial intelligence (AI) has led to increasing concerns about the capability of AI systems to make decisions and behave responsibly. Responsible AI (RAI) refers to the development and use of AI systems that benefit humans, society, and the environment while minimising the risk of negative consequences. To ensure responsible AI, the risks associated with AI systems' development and use must be identified, assessed and mitigated. Various AI risk assessment frameworks have been released recently by governments, organisations, and companies. However, it can be challenging for AI stakeholders to have a clear picture of the available frameworks and determine the most suitable ones for a specific context. Additionally, there is a need to identify areas that require further research or development of new frameworks, as well as updating and maintaining existing ones. To fill the gap, we present a mapping study of 16 existing AI risk assessment frameworks from the industry, governments, and non-government organizations (NGOs). We identify key characteristics of each framework and analyse them in terms of RAI principles, stakeholders, system lifecycle stages, geographical locations, targeted domains, and assessment methods. Our study provides a comprehensive analysis of the current state of the frameworks and highlights areas of convergence and divergence among them. We also identify the deficiencies in existing frameworks and outlines the essential characteristics of a concrete and connected framework AI risk assessment (C2AIRA) framework. Our findings and insights can help relevant stakeholders choose suitable AI risk assessment frameworks and guide the design of future frameworks towards concreteness and connectedness.

Blockchain technology has been integrated into diverse software applications by enabling a decentralised architecture design. However, the defects of on-chain algorithmic mechanisms, and tedious disputes and debates in off-chain communities may affect the operation of blockchain systems. Accordingly, blockchain governance has received great interest for supporting the design, use, and maintenance of blockchain systems, hence improving the overall trustworthiness. Although much effort has been put into this research topic, there is a distinct lack of consideration for blockchain governance from the perspective of software architecture design. In this study, we propose a pattern-oriented reference architecture for governance-driven blockchain systems, which can provide guidance for future blockchain architecture design. We design the reference architecture based on an extensive review of architectural patterns for blockchain governance in academic literature and industry implementation. The reference architecture consists of four layers. We demonstrate the components in each layer, annotating with the identified patterns. A qualitative analysis of mapping two concrete blockchain architectures, Polkadot and Quorum, on the reference architecture is conducted, to evaluate the correctness and utility of proposed reference architecture.

Federated learning is an emerging privacy-preserving AI technique where clients (i.e., organizations or devices) train models locally and formulate a global model based on the local model updates without transferring local data externally. However, federated learning systems struggle to achieve trustworthiness and embody responsible AI principles. In particular, federated learning systems face accountability and fairness challenges due to multistakeholder involvement and heterogeneity in client data distribution. To enhance the accountability and fairness of federated learning systems, we present a blockchain-based trustworthy federated learning architecture. We first design a smart contract-based data-model provenance registry to enable accountability. Additionally, we propose a weighted fair data sampler algorithm to enhance fairness in training data. We evaluate the proposed approach using a COVID-19 X-ray detection use case. The evaluation results show that the approach is feasible to enable accountability and improve fairness. The proposed algorithm can achieve better performance than the default federated learning setting in terms of the model’s generalization and accuracy.

Device failure detection is one of most essential problems in Industrial Internet of Things (IIoT). However, in conventional IIoT device failure detection, client devices need to upload raw data to the central server for model training, which might lead to disclosure of sensitive business data. Therefore, in this article, to ensure client data privacy, we propose a blockchain-based federated learning approach for device failure detection in IIoT. First, we present a platform architecture of blockchain-based federated learning systems for failure detection in IIoT, which enables verifiable integrity of client data. In the architecture, each client periodically creates a Merkle tree in which each leaf node represents a client data record, and stores the tree root on a blockchain. Furthermore, to address the data heterogeneity issue in IIoT failure detection, we propose a novel centroid distance weighted federated averaging (CDW_FedAvg) algorithm taking into account the distance between positive class and negative class of each client data set. In addition, to motivate clients to participate in federated learning, a smart contact-based incentive mechanism is designed depending on the size and the centroid distance of client data used in local model training. A prototype of the proposed architecture is implemented with our industry partner, and evaluated in terms of feasibility, accuracy, and performance. The results show that the approach is feasible, and has satisfactory accuracy and performance.