• Multiple deep reinforcement learning agents coordinate home automation devices. • User interactions are modeled as implicit feedback to generate agent rewards • The recommender system successfully adapts to changes in users’ daily routines. • Two domain-specific action selection strategies adapted for reinforcement learning systems in environments with the possibility of routine change. • A new metric based on the Hamming Score for evaluating predictions for smart devices with a composite state. The rapid expansion of the Internet of Things (IoT) has provided the technological foundation for smart homes, in which interconnected devices enable the environment to adapt to residents’ needs. Yet, this proliferation of IoT-enabled devices also introduces a significant challenge: managing and coordinating their multiple operational states, which extend far beyond a simple “switch on/switch off”. Aiming to automate the action process and anticipate user actions, this study proposes a state recommender system for actuator devices in smart homes, developed using deep reinforcement learning algorithms integrated with implicit feedback. This approach enables composite state encoding and coordination among multiple agents, thereby anticipating user needs and adapting to dynamic routines. In addition to using a dataset captured from a real-world IoT environment, we use a smart home simulator to generate two datasets based on three different routines and perform experiments with two reinforcement learning algorithms, Deep Q-Learning and Differential Semi-gradient n-step SARSA, benchmarking their performance against an online Supervised Learning baseline based on Behavior Cloning. Additionally, we tested these algorithms using two-state approaches: simple and composite. The results confirm the system’s successful adaptation to user routines across both approaches, underscoring its potential to enhance personalization in smart home environments.

ABSTRACT Context Smart home devices have become increasingly popular in modern households, powered by the Internet of Things (IoT) advances. The data generated by smart devices can provide valuable insights into users' behavior and preferences. By analyzing the data, one can understand how people interact with their homes, thus creating a “smart home profile”. To comprehend the complete IoT ecosystem dynamics of an intelligent environment, it is necessary to learn from each IoT device to predict its status in the future time. Nevertheless, dealing with real‐world IoT data structure requires considerable preprocessing tasks and the employment of classifiers that can learn multiple IoT inputs from a single IoT message. Objective Aware of these challenges, this paper proposes a novel methodology to process multi‐label IoT data and provide a comprehensive comparison of multi‐label classifiers for forecasting the status of smart devices, considering their efficiency and accuracy. Method We propose a data transformation method to preprocess the IoT data to be used by multi‐label classifiers. This method is based on real data structure. Results We evaluate our proposal in two real‐world scenarios and various multi‐label classifiers. The promising findings indicate that efficient classifiers can generate many correct predictions for a comprehensive IoT ecosystem in a small fraction of a second. Conclusions Our proposed data transformation can fit the context of prediction to smart homes and work with multi‐label classifiers to understand user behavior.