This paper extends the notion of learning algorithms and learning equilibriums from repeated games theory to stochastic games. We introduce Foolproof Cooperative Learning (FCL), an algorithm that converges to an equilibrium strategy that allows cooperative strategies in self-play setting while being not exploitable by selfish learners. By construction, FCL is a learning equilibrium for repeated symmetric games. We illustrate the behavior of FCL on symmetric matrix and grid games, and its robustness to selfish learners.

In this paper, we propose a novel setting for Inverse Reinforcement Learning (IRL), namely "Learning from a Learner" (LfL). As opposed to standard IRL, it does not consist in learning a reward by observing an optimal agent but from observations of another learning (and thus sub-optimal) agent. To do so, we leverage the fact that the observed agent’s policy is assumed to improve over time. The ultimate goal of this approach is to recover the actual environment’s reward and to allow the observer to outperform the learner. To recover that reward in practice, we propose methods based on the entropy-regularized policy iteration framework. We discuss different approaches to learn solely from trajectories in the state-action space. We demonstrate the genericity of our method by observing agents implementing various reinforcement learning algorithms. Finally, we show that, on both discrete and continuous state/action tasks, the observer’s performance (that optimizes the recovered reward) can surpass those of the observed agent.

The CoWriter activity involves a child in a rich and complex interaction where he has to teach handwriting to a robot. The robot must convince the child it needs his help and it actually learns from his lessons. To keep the child engaged, the robot must learn at the right rate, not too fast otherwise the kid will have no opportunity for improving his skills and not too slow otherwise he may loose trust in his ability to improve the robot' skills. We tested this approach in real pedagogic/therapeutic contexts with children in difficulty over repeated long sessions (40-60 min). Through 3 different case studies, we explored and refined experimental designs and algorithms in order for the robot to adapt to the troubles of each child and to promote their motivation and self-confidence. We report positive observations, suggesting commitment of children to help the robot, and their comprehension that they were good enough to be teachers, overcoming their initial low confidence with handwriting.

Measuring “how much the human is in the interaction” - the level of engagement - is instrumental in building effective interactive robots. Engagement, however, is a complex, multi-faceted cognitive mechanism that is only indirectly observable. This article formalizes with-me-ness as one of such indirect measures. With-me-ness, a concept borrowed from the field of Computer-Supported Collaborative Learning, measures in a well-defined way to what extent the human is with the robot over the course of an interactive task. As such, it is a meaningful precursor of engagement. We expose in this paper the full methodology, from real-time estimation of the human's focus of attention (relying on a novel, open-source, vision-based head pose estimator), to on-line computation of with-me-ness. We report as well on the experimental validation of this approach, using a naturalistic setup involving children during a complex robot-teaching task.

In this paper, we present an experiment in the context of a child-robot interaction where we study the influence of the child-robot spatial arrangement on the child's focus of attention and the perception of the robot's performance. In the “Co-Writer learning by teaching” activity, the child teaches a Nao robot how to handwrite. Usually only face-to-face spatial arrangements are tested in educational child robot interactions, but we explored two spatial conditions from Kendon's F-formation, the side-by-side and the face-to-face formations in a within subject experiment. We estimated the gaze behavior of the child and their consistency in grading the robot with regard to the robot's progress in writing. Even-though the demonstrations provided by children were not different between the two conditions (i.e. the robot's learning didn't differ), the results showed that in the side-by-side condition children tended to be more indulgent with the robot's mistakes and to give it better feedback. These results highlight the influence of experimental choices in child-robot interaction.