The adoption of Large Language Models (LLMs) for code generation in data science offers substantial potential for enhancing tasks such as data manipulation, statistical analysis, and visualization. However, the effectiveness of these models in the data science domain remains underexplored. This paper presents a controlled experiment that empirically assesses the performance of four leading LLM-based AI assistants-Microsoft Copilot (GPT-4 Turbo), ChatGPT (o1-preview), Claude (3.5 Sonnet), and Perplexity Labs (Llama-3.1-70b-instruct)-on a diverse set of data science coding challenges sourced from the Stratacratch platform. Using the Goal-Question-Metric (GQM) approach, we evaluated each model’s effectiveness across task types (Analytical, Algorithm, Visualization) and varying difficulty levels. Our statistical testing confirms that all models achieved success rates significantly above $50 \%$, demonstrating performance beyond chance. ChatGPT and Claude significantly exceeded the $60 \%$ threshold, but no model reached $70 \%$, indicating limitations in achieving higher accuracy. ChatGPT maintained consistent performance across difficulty levels, whereas Claude’s success varied with task complexity. Hypothesis testing indicates that task type does not significantly impact success rate overall. For analytical tasks, efficiency analysis shows no significant differences in execution times, though ChatGPT tended to be slower and less predictable despite high success rates. For visualization tasks, while similarity quality among LLMs is comparable, ChatGPT consistently delivered the most accurate outputs. This study provides a structured, empirical evaluation of LLMs in data science, delivering insights that support informed model selection tailored to specific task demands. Our findings establish a framework for future AI assessments, emphasizing the value of rigorous evaluation beyond basic accuracy measures.

Business processes are commonly represented by modelling languages, such as Event-driven Process Chain (EPC), Yet Another Workflow Language (YAWL), and the most popular standard notation for modelling business processes, the Business Process Model and Notation (BPMN). Most recently, chatbots, programs that allow users to interact with a machine using natural language, have been increasingly used for business process execution support. A recent category of chatbots worth mentioning is generative-based chatbots, powered by Large Language Models (LLMs) such as OpenAI’s Generative Pre-Trained Transformer (GPT) model and Google’s Pathways Language Model (PaLM), which are trained on billions of parameters and support conversational intelligence. However, it is not clear whether generative-based chatbots are able to understand and meet the requirements of constructs such as those provided by BPMN for process execution support. This paper presents a case study to compare the performance of prominent generative models, GPT and PaLM, in the context of process execution support. The research sheds light into the challenging problem of using conversational approaches supported by generative chatbots as a means to understand process-aware modelling notations and support users to execute their tasks.

This paper introduces the ‘GPT-in-the-loop’ approach, which seeks to investigate the reasoning capabilities of Large Language Models (LLMs) like Generative Pre-trained Transformers (GPT) within multiagent systems (MAS). Moving beyond traditional adaptive approaches that generally require long training processes, our framework employs GPT-4 to enhance problem-solving and explanation skills. To explore this approach, we apply it to a smart streetlight application in the Internet of Things (IoT) context, wherein each streetlight is controlled by an autonomous agent equipped with sensors and actuators, tasked with creating an energy-efficient lighting system. With the integration of GPT-4, these agents have shown enhanced decision-making and adaptability, without necessitating prolonged training. We compare this approach with both conventional neuroevolutionary methods and manually crafted solutions by software engineers, underscoring the potential of GPT-driven behavior in multiagent systems. It is important to note that these comparisons are preliminary, and further, more extensive testing is critical to determine the approach’s applicability across a wider range of MAS scenarios. Structurally, the paper delineates the incorporation of GPT into the agent-driven Framework for the Internet of Things (FIoT), details our proposed GPT-in-the-loop approach, presents comparative results within the IoT setting, and concludes with insights and prospective future directions.

There is an increasing interest in leveraging Large Language Models (LLMs) for managing structured data and enhancing data science processes. Despite the potential benefits, this integration poses significant questions regarding their reliability and decision-making methodologies. Our objective is to elucidate and express the factors and assumptions guiding GPT-4’s model selection recommendations. It highlights the importance of various factors in the model selection process, including the nature of the data, problem type, performance metrics, computational resources, interpretability vs accuracy, assumptions about data, and ethical considerations. We employ a variability model to depict these factors and use toy datasets to evaluate both the model and the implementation of the identified heuristics. By contrasting these outcomes with heuristics from other platforms, our aim is to determine the effectiveness and distinctiveness of GPT-4’s methodology. This research is committed to advancing our comprehension of AI decision-making processes, especially in the realm of model selection within data science. Our efforts are directed towards creating AI systems that are more transparent and comprehensible, contributing to a more responsible and efficient practice in data science.

Data science projects often involve various machine learning (ML) methods that depend on data, code, and models. One of the key activities in these projects is the selection of a model or algorithm that is appropriate for the data analysis at hand. ML model selection depends on several factors, which include data-related attributes such as sample size, functional requirements such as the prediction algorithm type, and nonfunctional requirements such as performance and bias. However, the factors that influence such selection are often not well understood and explicitly represented. This paper describes ongoing work on extending an adaptive variability-aware model selection method with bias detection in ML projects. The method involves: (i) modeling the variability of the factors that affect model selection using feature models based on heuristics proposed in the literature; (ii) instantiating our variability model with added features related to bias (e.g., bias-related metrics); and (iii) conducting experiments that illustrate the method in a specific case study to illustrate our approach based on a heart failure prediction project. The proposed approach aims to advance the state of the art by making explicit factors that influence model selection, particularly those related to bias, as well as their interactions. The provided representations can transform model selection in ML projects into a non ad hoc, adaptive, and explainable process.

Smart wearable devices for patient monitoring rely on batteries as energy-source for capturing vital signs, processing information locally, and transmitting data. The advantages of such solutions are providing mobility to users, connectivity to send data constantly, and low cost. These devices are wireless and must be tiny to be carried comfortably by the users. This fact restricts energy autonomy and requires frequent replacement or recharge of batteries. The highest energy cost is commonly attributed to transmissions in wireless devices, and several studies focused on communication and routing protocols to enhance energy efficiency in such solutions. However, researchers should give more attention to data acquisition of physiological sensors regarding energy efficiency in such solutions. In this preliminary study, we present the effects of a self-adaptive algorithm on the energy consumption of popular wearable physiological sensors. Our prototype is composed of an oximeter and a temperature sensor. Our experiments demonstrate that the self-adaptive procedure can save up to 80% energy consumption regarding the oximeter when monitoring stable patients at low risk and 51% in unstable patients. In addition, the temperature sensor can reach 97% of energy savings in the self-adaptive mode. The sensors’ data acquisition can present a superior energy cost than radio transmissions on such devices. In future work, we will explore the potential benefits of the algorithm in all main activities of our monitoring device.

Recent advances in software and artificial intelligence technologies have led to the increasing need to support the collaboration between humans and systems in various application domains. The growing capacity for systems has leveraged the power of building applications that have higher autonomy. However, a proper understanding of allocating tasks to either humans or machines is still lacking, and no suggestions are provided to support this allocation. Current approaches do not consider knowledge about the appropriate level of automation (LOA) in this collaboration and do not support adaptive automation, especially task assignments during the system’s operation. The knowledge about which factors affect the variability in human-system interaction LOA has not been explicitly captured. This paper presents a preliminary study that identifies the factors that influence levels of automation in autonomous systems and present the identified factors as a list. Identifying the factors that influence the level of autonomy of systems advances research in the design of autonomous systems by introducing an adaptive automation approach that can recommend levels of automation to support human-computer interactions. Modern systems must be prepared to identify, capture and process the significant volume and variety of data related to the factors that might influence the variability of systems’ behaviours.

Data science projects involve a machine learning (ML) process based on data, code, and models that change over time. For example, the datasets may increase in size and allow an ML model that requires larger datasets to be applied. However, the dynamic factors that influence model selection are not well understood and explicitly represented. This paper presents ongoing work on an adaptive method for ML model selection in big data science projects. The proposed method involves (i) identifying the factors that affect model selection based on heuristics proposed in the literature; and (ii) modeling the variability of these factors using a feature diagram and constraints that trigger adaptive reconfiguration, that is, changes in model selection due to changes in the variability factors. The applicability of the method is demonstrated through an illustrative use case. The proposed method can lead to an improved understanding of dynamic factors that influence model selection, how these factors explicitly affect the selection, and how the adaptive factors can be represented and automated. This improved understanding can result in a project model selection process that is less implicit and more efficient, more adaptive and explainable, and ultimately constitute a foundation for the creation of novel dynamic software product lines to support this process.

There is a lack of approaches and tools to support the development of projects in which humans and machines (e.g., machine learning algorithms) need to collaborate to achieve a specified goal. Specifically, given a set of software development tasks to develop a project collaboratively, how can these tasks be assigned to humans or machines to perform each task most efficiently and effectively? Such understanding is essential to support new methodologies for developing human-in-the-loop approaches in which machine learning automated procedures assist software developers in achieving their tasks. This paper describes our work in progress towards providing an approach to guide the assignment of tasks in developing human-in-the-loop big data (science) software development projects. The paper provides several contributions, including the provision of (i) a human-in-the-loop approach for the development of big data software development projects; (ii) the application of the approach to two case studies; (iii) a discussion of implications and research opportunities.

Internet of Things (IoT) applications have been deployed in several domains, including health care, smart cities, and agriculture. Because of the complex static and dynamic variability of the environment in which these applications are deployed, machine learning-based approaches have been used to support the design of IoT applications. In particular, an emergent approach involves using neural networks to enable IoT devices to learn to adapt their behavior based on the dynamics of the environment. Designing IoT systems is already challenging because of the autonomy and concurrency inherent in distributed physical systems. Moreover, neural networks systems have particular characteristics, such as dynamism, adaptability, and generalization, that make it necessary to adapt the traditional software development lifecycle to satisfy the requirements of these systems. In this paper, we describe our proposed approach to support the engineering of IoT neural network-based systems. Our approach considers a lifecycle supporting the integration of IoT system development tasks with particular ANN tasks, as model requirements and feature engineering. In addition, the paper includes the provision of the application of the approach to a case study and conclusive remarks.

Emergent software applications are increasingly becoming (self-)adaptive and autonomous. Further, Internet of Things (IoT) applications increasingly involve data analytics. The introduction of neural networks in IoT systems has enabled a new generation of applications capable of performing complex sensing and actuation analysis tasks that were not previously possible with other approaches. A key component in the development of these systems is the ability to represent data analytics variability, which captures the ways in which the system can adapt in terms of the data analysis at design and run times. Although variability has been explored in the domain of software product lines (SPLs), data analytics variability in IoT neural network-based systems still seems to be poorly understood and needs to be investigated appropriately. In this paper, we introduce an approach to capture data analytics variability in IoT neural network-based systems (IoTNNSs). The approach represents several types of variability inherent in the development of these analytics systems, including those related to the application context, behavior, quality attributes, IoT devices, and neural networks.

Data analysis involves the use of a wide variety of systems and libraries to support the exploration and development of models that can uncover valuable patterns and enable individuals and businesses to draw informed insights. However, efforts towards the automation of the ML-based data analysis modeling process faces numerous challenges. In this paper, we describe our ongoing work towards the automation of the data analysis modeling phase based on a variability-aware approach. This approach involves capturing the variabilities through feature models, designing an automated framework to support the analysis, and developing use cases. The work advances the state of the art in the development of methods and tools to support the automation of ML-based data analysis.

Embodied agents is a term used to denote intelligent agents, which are a component of devices belonging to the Internet of Things (IoT) domain. Each agent is provided with sensors and actuators to interact with the environment, and with a `controller' that usually contains an artificial neural network (ANN). In previous publications, we introduced three software approaches to design, implement and test IoT embodied agents. In this paper, we propose a reference model based on statecharts that offers abstractions tailored to the development of IoT applications. The model represents embodied agents that are controlled by neural networks. Our model includes the ANN training process, represented as a reconfiguration step such as changing agent features or neural net connections. Our contributions include the identification of the main characteristics of IoT embodied agents, a reference model specification based on statecharts, and an illustrative application of the model to support autonomous street lights. The proposal aims to support the design and implementation of IoT applications by providing high-level design abstractions and models, thus enabling the designer to have a uniform approach to conceiving, designing and explaining such applications.

Several papers have recently contained reports on applying machine learning (ML) to the automation of software engineering (SE) tasks, such as project management, modeling and development. However, there appear to be no approaches comparing how software engineers fare against machine-learning algorithms as applied to specific software development tasks. Such a comparison is essential to gain insight into which tasks are better performed by humans and which by machine learning and how cooperative work or human-in-the-loop processes can be implemented more effectively. In this paper, we present an empirical study that compares how software engineers and machine-learning algorithms perform and reuse tasks. The empirical study involves the synthesis of the control structure of an autonomous streetlight application.

Agent-based Internet of Things (IoT) applications have recently emerged as applications that can involve sensors, wireless devices, machines and software that can exchange data and be accessed remotely. Such applications have been proposed in several domains including health care, smart cities and agriculture. However, despite their increased adoption, deploying these applications in specific settings has been very challenging because of the complex static and dynamic variability of the physical devices such as sensors and actuators, the software application behavior and the environment in which the application is embedded. In this paper, we propose a modeling approach for IoT analytics based on learning embodied agents (i.e. situated agents). The approach involves: (i) a variability model of IoT embodied agents; (ii) feedback evaluative machine learning; and (iii) reconfiguration of a group of agents in accordance with environmental context. The proposed approach advances the state of the art in that it facilitates the development of Agent-based IoT applications by explicitly capturing their complex and dynamic variabilities and supporting their self-configuration based on an context-aware and machine learning-based approach.