The Model AI Assignments session seeks to gather and dis- seminate the best assignment designs of the Artificial In- telligence (AI) Education community. Recognizing that as- signments form the core of student learning experience, we here present abstracts of five AI assignments from the 2024 session that are easily adoptable, playfully engaging, and flexible for a variety of instructor needs. Assignment spec- ifications and supporting resources may be found at http://modelai.gettysburg.edu.

When robots have multiple tasks to perform, they must determine the order in which to complete them. Interleaving tasks is efficient for the robot trying to finish its to-do list, but it may be less satisfying for a human whose request was delayed in favor of schedule efficiency. Following online research that examined delays with various motivations, we created two in-person studies in which participants’ tasks were impacted by the robot’s other tasks. In the first, participants either requested a task for the robot to complete on their behalf or watched the robot performing tasks for other people. We measured how their opinions changed depending on whether their task’s completion was delayed due to another participant’s task or they were observing without a task of their own. In the second, participants had a robot walk them to an office and became delayed as the robot detoured to another location. We measured how opinions of the robot changed depending on who requested the detour task and the length of the detour. Overall, participants positively viewed task interleaving as long as the delay and inconvenience imposed by someone else’s task were small and the task was well-justified. Also, observers often had lower opinions of the robot than participants who requested tasks, highlighting a concern for online research.

A majority of the courses on autonomous systems focus on robotics, despite the growing use of autonomous agents in a wide spectrum of applications, from smart homes to intelligent traffic control. Our goal in designing a new senior-level undergraduate course is to teach the integration of a variety of AI techniques in uncertain environments, without the dependence on topics such as robotic control and localization. We chose the application of an autonomous greenhouse to frame our discussions and our student projects because of the greenhouse's self-contained nature and objective metrics for successfully growing plants. We detail our curriculum design, including lecture topics and assignments, and our iterative process for updating the course over the last four years. Finally, we present some student feedback about the course and opportunities for future improvement.

Recent work has suggested that a robot that in-terrupts assigned tasks for the sake of curiosity is perceived as less competent, but that communicating acknowledgment of the curious behavior can mitigate some of those feelings [1]. In real-world situations, there are many reasons why a robot's task could be interrupted in favor of another. For example, a robot handling requests for tasks from people in different locations could navigate more efficiently if it interleaves those tasks, but it ideally would not do so at the expense of the users' perceptions of the robot. In order to understand the impact of different task interleaving patterns on human perceptions of a robot's behavior, we performed a study in which a robot performed a delivery task and an investigative task, interleaving them in various ways. The participants were told either that the investigative task was motivated by a request from another person, motivated by curiosity, or they received no information about why the robot performed the action. While participants acknowledged that interleaving tasks should be allowed, they rated the robot as more competent when its tasks were not interleaved. They were most receptive to interleaving when they knew the investigative task was for another person and less receptive to long task detours away from the delivery route, especially when the inspection task was motivated by curiosity.

The success of data scientists in developing machine learning models is contingent on an iterative development process for detecting patterns in data, finding and extracting useful features, and maximizing their model’s performance. However, it is often the case that they struggle during model development and become stuck and unable to make significant progress. We collected qualitative and quantitative data from the workflow of data scientists that allow us to learn from and examine such moments of stuckness. We used this data to develop a model for predicting stuckness based on real-time indicators, such as code artifacts, and then used the model to develop an innovative algorithm that determines precisely when a potential stuckness intervention should occur: as close as possible to the beginning of actual stuckness. Our algorithm’s performance indicates the potential efficacy of predicting data scientist stuckness algorithmically under real-world circumstances and for real-world needs.

Data analysts often tediously create visualization sequences to derive insights about what they see. While recent AI-driven approaches generate sequences to optimize visualization appeal and individual user preferences, extended cognitive fit theory suggests that expertise and insight type will affect the visualizations that analysts prefer. To investigate the role of expertise on insight generation from visualization sequences, we asked data scientists and accountants to report their insights as they investigated two business datasets. We found that both groups frequently followed the visualization sequences in order. However, expertise played a role in predicting the types of visualizations that each group chose to visit when they had finished the sequence but had time remaining. We also found significant interaction effects of visualization type, insight type, and expertise when assessing the numbers of insights generated per participant. Based on these results, we recommend that AI-driven data visualization tools should incorporate expertise as a feature for predicting new visualizations to produce.

As robots are deployed to work in our environments, we must build appropriate expectations of their behavior so that we can trust them to perform their jobs autonomously as we attend to other tasks. Many types of explanations for robot behavior have been proposed, but they have not been fully analyzed for their impact on aligning expectations of robot paths for navigation. In this work, we evaluate several types of robot navigation explanations to understand their impact on the ability of humans to anticipate a robot’s paths. We performed an experiment in which we gave participants an explanation of a robot path and then measured (i) their ability to predict that path, (ii) their allocation of attention on the robot navigating the path versus their own dot-tracking task, and (iii) their subjective ratings of the robot’s predictability and trustworthiness. Our results show that explanations do significantly affect people’s ability to predict robot paths and that explanations that are concise and do not require readers to perform mental transformations are most effective at reducing attention to the robot.

People take to social media to share their thoughts, joys, and sorrows. A recent popular trend has been to support and mourn people and pets that have died as well as other objects that have suffered catastrophic damage. As several popular robots have been discontinued, including the Opportunity Rover, Jibo, and Kuri, we are interested in how language used to mourn these robots compares to that to mourn people, animals, and other objects. We performed a study in which we asked participants to categorize deidentified Twitter reactions as referencing the death of a person, an animal, a robot, or another object. Most reactions were labeled as being about humans, which suggests that people use similar language to describe feelings for animate and inanimate entities. We used a natural language toolkit to analyze language from a larger set of tweets. A majority of tweets about Opportunity included second-person ("you") and gendered third-person pronouns (she/he versus it), but terms like "R.I.P" were reserved almost exclusively for humans and animals. Our findings suggest that people verbally mourn robots similarly to living things, but reserve some language for people.

Data science is a growing field at the intersection of mathematics, computer science, and domain expertise. Like many universities that are building data science degree programs for undergraduates, our small, liberal-arts university saw increasing opportunities in the region and decided to build a data science degree from the ground up, without a pre-existing computer science (CS) department to leverage for courses or culture. We designed and implemented an academically-demanding curriculum that combined mathematics, information systems, and new data science courses, and that also encouraged and supported student success. Each introductory course included active learning design to engage students. To increase retention, all major courses included assignments designed to build skills but also student confidence in their ability to learn challenging technical topics. Outside of the classroom, we created opportunities for professional advancement and developed a technical culture at the university. We will share our approach, course highlights, and lessons learned from building such a curriculum at an institution without a CS department.

Artificial Intelligence (AI) has had a transformational impact on Human-Robot Interaction (HRI) research over the past decade, enabling work in HRI to develop and investigate robots that can operate autonomously in far more challenging environments and far more complex scenarios than was possible ever before.Beyond laboratory studies, robots that explicitly interact with people as part of their functionality are increasingly being developed, productized, and deployed throughout the world, enabling ecologically valid ethnographic studies of interactions between humans and robots.These advances have been fueled by enabling technologies across many subfields of AI including machine learning, computer vision, task and motion planning, natural language understanding, and dialogue systems.It is not, however, the case that AI research produced polished, ready-off-the-shelf tools that researchers could pick up and effortlessly use to build their envisioned autonomous robot.Rather, the shift has been due to a new, hybrid approach to human-centered robotics research, facilitated by HRI researchers who acquired deep technical skill sets and an influx of AI researchers applying their expertise to HRI problems.More interdiscplinary research teams consisting of formerly AI and HRI researchers also formed, resulting in a vibrant sub-community at the intersection of AI and HRI who came together at the AAAI Fall Symposium on AI for Human-Robot Interaction for the last 4 years.This special issue was encouraged by the continued success and overwhelming popularity of this symposium.Our goal is to exemplify this community's mature, high-quality, and original work, establishing T-HRI as a premier venue for work at the intersection of AI and HRI.Research at this intersection is particularly challenging due to the very need for interdiscplinary, multi-faceted skill sets.AI-HRI researchers need to both innovate in computational techniques and

While recent advances in computer vision have caused object recognition rates to spike, there is still much room for improvement. In this paper, we develop an algorithm to improve object recognition by integrating human-generated contextual information with vision algorithms. Specifically, we examine how interactive systems such as robots can utilize two types of context information--verbal descriptions of an environment and human-labeled datasets. We propose a re-ranking schema, MultiRank, for object recognition that can efficiently combine such information with the computer vision results. In our experiments, we achieve up to 9.4% and 16.6% accuracy improvements using the oracle and the detected bounding boxes, respectively, over the vision-only recognizers. We conclude that our algorithm has the ability to make a significant impact on object recognition in robotics and beyond.

People form beliefs about intentions and preferences of robots as they observe robot movement. However, robots rarely optimize their movement to allow people to easily determine state preferences. In this work, we define critical points along robot trajectories that convey information about state preferences: inflection points are changes in direction and compromise points are the relative proportion of preferred states to non-preferred ones. We contribute an approach for automatically generating trajectory demonstrations with specified critical points, and test observers' abilities to understand and generalize our robot's preferences based on our generated demonstrations. Our results show that inflection points helped participants understand state preference ordering and allowed them to more accurately predict paths through new environments, while compromise points hindered understanding. We conclude that robots should evaluate their trajectories for critical points to increase human observer understanding.

In order to be successfully integrated into human-populated environments, mobile robots need to express relevant information about their state to the outside world. In particular, animated lights are a promising way to express hidden robot state information such that it is visible at a distance. In this work, we present an online study to evaluate the effect of robot communication through expressive lights on people's understanding of the robot's state and actions. In our study, we use the CoBot mobile service robot with our light interface, designed to express relevant robot information to humans. We evaluate three designed light animations on three corresponding scenarios for each, for a total of nine scenarios. Our results suggest that expressive lights can play a significant role in helping people accurately hypothesize about a mobile robot's state and actions from afar when minimal contextual clues are present. We conclude that lights could be generally used as an effective non-verbal communication modality for mobile robots in the absence of, or as a complement to, other modalities.

As humans and robots collaborate together on spatial tasks, they must communicate clearly about the objects they are referencing. Communication is clearer when language is unambiguous which implies the use of spatial references and explicit perspectives. In this work, we contribute two studies to understand how people instruct a partner to identify and pick up objects on a table. We investigate spatial features and perspectives in human spatial references and compare word usage when instructing robots vs. instructing other humans. We then focus our analysis on the clarity of instructions with respect to perspective taking and spatial references. We find that only about 42% of instructions contain perspective-independent spatial references. There is a strong correlation between participants' accuracy in executing instructions and the perspectives that the instructions are given in, as well between accuracy and the number of spatial relations that were required for the instruction. We conclude that sentence complexity (in terms of spatial relations and perspective taking) impacts understanding, and we provide suggestions for automatic generation of spatial references.

With a growing number of robots performing autonomously without human intervention, it is difficult to understand what the robots experience along their routes during execution without looking at execution logs. Rather than looking through logs, our goal is for robots to respond to queries in natural language about what they experience and what routes they have chosen. We propose verbalization as the process of converting route experiences into natural language, and highlight the importance of varying verbalizations based on user preferences. We present our verbalization space representing different dimensions that verbalizations can be varied, and our algorithm for automatically generating them on our CoBot robot. Then we present our study of how users can request different verbalizations in dialog. Using the study data, we learn a language model to map user dialog to the verbalization space. Finally, we demonstrate the use of the learned model within a dialog system in order for any user to request information about CoBot's route experience at varying levels of detail.

While big data analytics continue to grow in popularity among companies and organizations, their large-scale analytic implementations are often completed by software developers with little or no formal training in machine learning or data analysis. These developers are skilled at writing code but they do not have the understanding of the data analytics process to be efficient or necessarily accurate at it. These developers use processes and tools that are often ad hoc and incomplete as they learn by doing. We followed a development team through two analytics development cycles and analyzed their interactions with their data and tools. In this paper, we first describe the tools the developers used and then present concrete opportunities for the big data community to create tools that empower these developers to build more accurate analytics more efficiently.

Indoor autonomous mobile service robots can overcome their hardware and potential algorithmic limitations by asking humans for help. In this work, we focus on mobile robots that need human assistance at specific spatially-situated locations (e.g., to push buttons in an elevator or to make coffee in the kitchen). We address the problem of what the robot should do when there are no humans present at such help locations. As the robots are mobile, we argue that they should plan to proactively seek help and travel to offices or occupied locations to bring people to the help locations. Such planning involves many trade-offs, including the wait time at the help location before seeking help, and the time and potential interruption to find and displace someone in an office. In order to choose appropriate parameters to represent such decisions, we first conduct a survey to understand potential helpers' travel preferences in terms of distance, interruptibility, and frequency of providing help. We then use these results to contribute a decision-theoretic algorithm to evaluate the possible choices in offices and plan where to proactively seek help. We demonstrate that our algorithm aims to minimize the number of office interruptions as well as task completion time.

In this video we briefly illustrate the progress and contributions made with our mobile, indoor, service robots CoBots (Collaborative Robots), since their creation in 2009. Many researchers, present authors included, aim for autonomous mobile robots that robustly perform service tasks for humans in our indoor environments. The efforts towards this goal have been numerous and successful, and we build upon them. However, there are clearly many research challenges remaining until we can experience intelligent mobile robots that are fully functional and capable in our human environments.

AI agents including robots often use reward functions to evaluate tradeoffs between different states and actions and to determine optimal policies. We are particularly interested in reward functions that can be decomposed into an additive sum of subrewards that are computed on independent subproblems or features of the state space. If these subrewards capture different reward metrics, such as user satisfaction and task completion time, it is unclear how to scale the subrewards in the reward function to produce an appropriate policy. In this work, we propose and evaluate a novel Monte Carlo method for learning the scaling factors of subrewards, in which the training elicits humans' preferences between two state-action scenarios. Because the algorithm elicits preferences over explicit scenarios, it is less susceptible to human error than previous elicitation approaches. The preferences are used to generate a set of inequalities over the scaling factors that we solve efficiently using a linear program. We show that our algorithm asks for a number of preferences proportional to log of the number of scaling factor hypotheses used in the Monte Carlo method.

When mobile robots perform tasks in environments with humans, it seems appropriate for the robots to rely on such humans for help instead of dedicated human oracles or supervisors. However, these humans are not always available nor always accurate. In this work, we consider human help to a robot as concretely providing observations about the robot's state to reduce state uncertainty as it executes its policy autonomously. We model the probability of receiving an observation from a human in terms of their availability and accuracy by introducing Human Observation Providers POMDPs (HOP-POMDPs). We contribute an algorithm to learn human availability and accuracy online while the robot is executing its current task policy. We demonstrate that our algorithmis effective in approximating the true availability and accuracy of humans without depending on oracles to learn, thus increasing the tractability of deploying a robot that can occasionally ask for help.

The ability to obtain accurate observations while navigating in uncertain environments is a difficult challenge in deploying robots. Robots have relied heavily on human supervisors who are always available to provide additional observations to reduce uncertainty. We are instead interested in taking advantage of humans who are already in the environment to receive observations. The challenge is in modeling these humans' availability and higher costs of interruption to determine when to query them during navigation. In this work, we introduce a Human Observation Provider POMDP framework (HOP-POMDP), and contribute new algorithms for planning and executing with HOP-POMDPs that account for the differences between humans and other probabilistic sensors that provide observations. We compare optimal HOP-POMDP policies that plan for needing humans' observations with oracle POMDP policies that do not take human costs and availability into account. We show in benchmark tests and real-world environments that the oracle policies match the optimal HOP-POMDP policy 60% of the time, and can be used in cases when humans are likely to be available on the shortest paths. However, the HOP-POMDP policies receive higher rewards in general as they take into account the possibility that a human may be unavailable. HOP-POMDP policies only need to be computed once prior to the deployment of the robot, so it is feasible to precompute and use in practice.

We present a study that evaluates the effectiveness of augmenting on-screen instructions with micro-projection for manual task guidance unlike prior work, which replaced screen instructions with alternative modalities (e.g., head-mounted displays). In our study, 30 participants completed 10 trials each of 11 manual tasks chosen to represent a set of common task-components (e.g., cutting, folding) found in many everyday activities such as crafts, cooking, and hobby electronics. Fifteen participants received only on-screen instructions, and 15 received both on-screen and micro-projected instructions. In contrast to prior work, which focused only on whole tasks, our study examines the benefit of augmenting common task instructions. The augmented instructions improved participants' performance overall; however, we show that in certain cases when projected guides and physical objects visually interfered, projected elements caused increased errors. Our results demonstrate that examining effectiveness at an instruction level is both useful and necessary, and provide insight into the design of systems that help users perform everyday tasks.

We present two studies that evaluate the accuracy of human responses to an intelligent agent's data classification questions. Prior work has shown that agents can elicit accurate human responses, but the applications vary widely in the data features and prediction information they provide to the labelers when asking for help. In an initial analysis of this work, we found the five most popular features, namely uncertainty, amount and level of context, prediction of an answer, and request for user feedback. We propose that there is a set of these data features and prediction information that maximizes the accuracy of labeler responses. In our first study, we compare accuracy of users of an activity recognizer labeling their own data across the dimensions. In the second study, participants were asked to classify a stranger's emails into folders and strangers' work activities by interruptibility. We compared the accuracy of the responses to the users' self-reports across the same five dimensions. We found very similar combinations of information (for users and strangers) that led to very accurate responses as well as more feedback that the agents could use to refine their predictions. We use these results for insight into the information that help labelers the most.

Asking questions is an inevitable part of collaborative interactions between humans and robots. However, robotics novices may have difficulty answering the robots' questions if they do not understand what the robot is asking. We are particularly interested in whether robots can supplement their questions with information about their state in a manner that increases the accuracy of human responses. In this work, we design and carefully analyze a human-robot collaborative task experiment to measure humans' responses and accuracies to different amounts of supplemental information. We vary the content of the questions along four dimensions of the robot state, namely uncertainty, context, predictions, and feature selection. Based on our results, we contribute guidelines on the effective combination of the four dimensions, under the assumption that the robot has no limitations on generating question context. Finally, we validate our guidelines against educated recommendations from the HRI community.

Valerie the roboceptionist is the most recent addition to Carnegie Mellon's social robots project. A permanent installation in the entranceway to Newell-Simon hall, the robot combines useful functionality - giving directions, looking up weather forecasts, etc. - with an interesting and compelling character. We are using Valerie to investigate human-robot social interaction, especially long-term human-robot "relationships". Over a nine-month period, we have found that many visitors continue to interact with the robot on a daily basis, but that few of the individual interactions last for more than 30 seconds. Our analysis of the data has indicated several design decisions that should facilitate more natural human-robot interactions.