Accurately predicting human behaviors is crucial for mobile robots operating in human-populated environments. While prior research primarily focuses on predicting actions in single-human scenarios from an egocentric view, several robotic applications require understanding multiple human behaviors from a third-person perspective. To this end, we present CAMP-VLM (Context-Aware Multi-human behavior Prediction): a Vision Language Model (VLM)-based framework that incorporates contextual features from visual input and spatial awareness from scene graphs to enhance prediction of humans-scene interactions. Due to the lack of suitable datasets for multi-human behavior prediction from an observer view, we perform fine-tuning of CAMP-VLM with synthetic human behavior data generated by a photorealistic simulator, and evaluate the resulting models on both synthetic and real-world sequences to assess their generalization capabilities. Leveraging Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO), CAMP-VLM outperforms the best-performing baseline by up to 66.9% in prediction accuracy.

Recent advancements in Large Language Models (LLMs) have sparked a revolution across many research fields. In robotics, the integration of common-sense knowledge from LLMs into task and motion planning has drastically advanced the field by unlocking unprecedented levels of context awareness. Despite their vast collection of knowledge, large language models may generate infeasible plans due to hal-lucinations or missing domain information. To address these challenges and improve plan feasibility and computational efficiency, we introduce DELTA, a novel LLM-informed task planning approach. By using scene graphs as environment representations within LLMs, DELTA achieves rapid generation of precise planning problem descriptions. To enhance planning performance, DELTA decomposes long-term task goals with LLMs into an autoregressive sequence of sub-goals, enabling automated task planners to efficiently solve complex problems. In our extensive evaluation, we show that DELTA enables an efficient and fully automatic task planning pipeline, achieving higher planning success rates and significantly shorter planning times compared to the state of the art. Project webpage: https://delta-llm.github.io/

Current buildings often operate in a relatively naïve manner, leading to inefficient energy use. This presents an opportunity to enhance efficiency of buildings’ operations by leveraging the advancements in the Internet of Things (IoT) and Artificial Intelligence (AI). AI planning, in particular, offers robust mechanisms for coordinating building operations while satisfying sustainability goals. However, two key obstacles hinder the application of AI planning in this context: the strong requirement for planning action models in a standardised syntax, such as the Planning Domain Definition Language (PDDL), and the lack of adequate tools for developing such models when planning experts and readily available plan traces are scarce. We introduce a novel approach for learning action models from IoT data in building environments semi-automatically. Our work is multifaceted in that it also supports the engineering process by presenting a reference model, a formalisation of the learning task and its correspondence to concept learning, a three-step method to accomplishing this task, and the method’s implementation in a multi-layered component-based system. We validate our approach on a case study using existing data from actual houses. The results show that the approach supports the learning of highly accurate PDDL action models that are capable of successfully solving planning problems, laying a pathway towards more intelligent and sustainable building management.

Given the prominence of AI planning in research and industry, the development of AI planning software and its integration into production architectures are becoming important. However, building and managing planning software is a complex and expertise-dependent process without methodological support that would ensure AI planning applications have high quality and industrial strength. To that end, we propose a lifecycle for developing AI planning systems that consists of ten phases related to the design, development, and operation of planning systems.

ChatGPT recently attracted vast attention in and outside the research community for its conversational abilities that mimic human ones exceptionally well.At the heart of systems like ChatGPT are Large Language Models (LLM).These models, rooted in deep neural networks, have the ability to predict the next textual token in a series of tokens based on statistical occurrences in extremely large data sets [1].When the models are sufficiently big and well-tuned, one observes the "unreasonable effectiveness of data" [2] in how the system generates perfectly intelligible and believable sentences.Such ability to have human-like conversations with a software system is both stunning for the quality of the conversation and mind-blowing in terms of the potential impact on society and the job market in particular [3,4].There is controversy on whether or not such systems manifest forms of artificial intelligence.Researchers at Microsoft, for instance, attribute signs of intelligence to the current fourth version of Generative Pre-trained Transformer (GPT-4), which is in development at the time of writing [5].Some authors have successfully solved Theory of Mind tasks using such tools.Kosinski reports a success rate of 95% using GPT-4 in solving false-belief tasks.Other authors are more careful with the excessive anthropomorphization of ChatGPT-like systems [6,7].What is sure is that the embedding of an LLM into a system makes it a very powerful tool.Of interest to us in this editorial is the LLM capability to generate programs [8] and its potential impact on Service-Oriented Computing and Applications.The problem of automated service composition is central to the field of Service-Oriented Computing and Applications [9].Seamless, unsupervised, automated composition of services available on a network is a potent way to build adaptive information systems.It is the idea that one can execute any task relying on multiple, loosely coupled services, pos-

Building managers need effective tools to improve occupants’ experiences considering constraints of energy efficiency. Current building management systems are limited to coordinating device services in simple and prefixed situations. Think of an office with lights offering services, such as turn on a light, which are invoked by the system to automatically control the lights. In spite of the evident potential for energy saving, the office occupants often end up in the dark, they have too much light when working with computers, or unnecessary lights are turned on. The office is thus not aware of the occupants’ presence nor anticipates their activities. Our proposal is to coordinate services while anticipating occupant activities with sufficient accuracy. Finding and composing services that will support occupant activities is however a complex problem. The high number of services, the continuous transformation of buildings, and the various building standards imply a search through a vast number of possible contextual situations every time occupants perform activities. Our solution to this building coordination problem is based on Hierarchical Task Network (HTN) planning in combination with activity recognition. While HTN planning provides powerful means for composing services automatically, activity recognition is needed to identify occupant activities as soon as they occur. The output of this combination is a sequence of services that needs to be executed under the uncertainty of building environments. Our solution supports continuous context changes and service failures by using an advanced orchestration strategy. We design, implement and deploy a system in two cases, namely offices and a restaurant, in our own office building at the University of Groningen. We show energy savings in the order of 80% when compared to manual control in both cases, and 60% when compared to using only movement sensors. Moreover, we show that one can save a figure of €600 annually for the electricity costs of the restaurant. We use a survey to evaluate the experience of restaurant occupants. The majority of them are satisfied with the solution and find it useful. Finally, the technical evaluation provides insights into the efficiency of our system.

We propose the use of HTN planning for risk-sensitive planning domains. We suggest utility functions that reflect the risk attitude of compound tasks, and adapt a best-first search algorithm to take such utilities into account.