Attractive images or videos are the visual backbones of journalism and social media to gain the user's attention. From trailers to teaser images to image galleries, appealing visuals have only grown in importance over the years. However, selecting eye-catching shots from a video or the perfect image from large image collections is a challenging and time-consuming task. We present our tool that can assess image and video content from an aesthetic standpoint. We discovered that it is possible to perform such an assessment by combining expert knowledge with data-driven information. We combine the relevant aesthetic features and machine learning algorithms into an aesthetics retrieval system, which enables users to sort uploaded visuals based on an aesthetic score and interact with additional photographic, cinematic, and person-specific features.

Knowledge of users’ affective states can improve their interaction with smartphones by providing more personalized experiences (e.g., search results and news articles). We present an affective state classification model based on data gathered on smartphones in real-world environments. From touch events during keystrokes and the signals from the inertial sensors, we extracted two-dimensional heat maps as input into a convolutional neural network to predict the affective states of smartphone users. For evaluation, we conducted a data collection in the wild with 82 participants over 10 weeks. Our model accurately predicts three levels (low, medium, high) of valence (AUC up to 0.83), arousal (AUC up to 0.85), and dominance (AUC up to 0.84). We also show that using the inertial sensor data alone, our model achieves a similar performance (AUC up to 0.83), making our approach less privacy-invasive. By personalizing our model to the user, we show that performance increases by an additional 0.07 AUC.

Recommender systems (RSs) completely rely on the knowledge of training information to generate recommendations. However, due to privacy, ownership, and protection of users’ information, such training information is not easily accessible or shared with an RS. Moreover, with recent regulations in privacy laws (e.g, GDPR), collecting user preferences and perform centralized training may not be feasible. Federated Learning (FL) is a form of machine learning technique where the goal is to learn a high-quality recommendation model without never directly accessing raw training data. In this work, we specifically focus on situations where multiple stakeholders (referred to as corporate companies like e-commerce business partners, hospitals, banks, news media publishers) participate in federated learning to build a shared recommendation model. We performed offline experiments by simulating a real federated learning setup and investigated the benefits federated learning brings to stakeholders in terms of ranking compared to an RS model trained without participating in federated learning. Our experimental results reveal that stakeholders can significantly benefit from federated learning to generate accurate recommendations. Moreover, we also study the use and benefits of federated learning in situations when there are not enough preferences available for users.

Federated Learning (FL) allows to collaboratively build machine learning models between different entities without the need for sharing or gathering the data. In FL, typically there is a global server and a set of clients (stakeholders) to build shared machine learning models. In contrast to distributed machine learning, the controller of the training process (here the global server) never sees the data of the stakeholders participating in FL. Every stakeholder owns his own data and doesn't share it. During the training and learning process, only the model updates (e.g. gradients) are shared. To our best of knowledge, we did not find a publicly available practical federated learning framework for stakeholders. We have built a framework that enables FL for a small number of stakeholders. In the paper, we describe the framework architecture, communication protocol, and algorithms. Our framework is open-sourced and it is easy to set up for stakeholders and ensures that no private information is leaked during the training process.

Gaining awareness of the user's affective states enables smartphones to support enriched interactions that are sensitive to the user's context. To accomplish this on smartphones, we propose a system that analyzes the user's text typing behavior using a semi-supervised deep learning pipeline for predicting affective states measured by valence, arousal, and dominance. Using a data collection study with 70 participants on text conversations designed to trigger different affective responses, we developed a variational auto-encoder to learn efficient feature embeddings of two-dimensional heat maps generated from touch data while participants engaged in these conversations. Using the learned embedding in a cross-validated analysis, our system predicted three levels (low, medium, high) of valence (AUC up to 0.84), arousal (AUC up to 0.82), and dominance (AUC up to 0.82). These results demonstrate the feasibility of our approach to accurately predict affective states based only on touch data.

The strong interest children show for mobile robots makes these devices potentially powerful to teach programming. Moreover, the tangibility of physical objects and the sociability of interacting with them are added benefits. A key skill that novices in programming have to acquire is the ability to mentally trace program execution. However, because of their embodied and real-time nature, robots make the mental tracing of program execution difficult.

The adaptivity of intelligent tutoring systems relies on the accuracy of the student model and the design of the instructional policy. Recently an instructional policy has been presented that is compatible with all common student models. In this work we present the next step towards a universal instructional policy. We introduce a new policy that is applicable to an even wider range of student models including DBNs modeling skill topologies and forgetting. We theoretically and empirically compare our policy to previous policies. Using synthetic and real world data sets we show that our policy can effectively handle wheel-spinning students as well as forgetting across a wide range of student models.

In our previous research, we showed that students using the educational robot Thymio and its visual programming environment were able to learn the important computer-science concept of event-handling. This paper extends that work by integrating augmented reality (AR) into the activities. Students used a tablet that displays in real time the event executed on the robot. The event is overlaid on the tablet over the image from a camera, which shows the location of the robot when the event was executed. In addition, visual feedback (FB) was implemented in the software. We developed a novel video questionnaire to investigate the performance of the students on robotics tasks. Data were collected comparing four groups: AR+FB, AR+non-FB, non-AR+FB, non-AR+non-FB. The results showed that students receiving feedback made significantly fewer errors on the tasks. Those using AR made fewer errors, but this improvement was not significant, although their performance improved. Technical problems with the AR hardware and software showed where improvements are needed.

We present HeapCraft: an open-source suite of interactive data exploration and visualization tools that allows researchers, server administrators and game designers to analyze and potentially influence player behavior in Minecraft. Our framework includes a telemetry system, several tools for visualizing and representing the collected data, and tools for modifying the game experience in controlled ways. Measures that we use to quantify and visualize player behavior and collaboration have been derived from a large data set containing 3451 player-hours from 908 players and 43 different servers. HeapCraft has been demonstrated on a variety of tasks including player behavior classification, as well as quantifying and improving collaboration of players on Minecraft servers. HeapCraft is freely available and serves to democratize game analytics for the Minecraft community at large.