Existing calibration algorithms address the problem of covariate shift via unsupervised domain adaptation. However, these methods suffer from the following limitations: 1) they require unlabeled data from the target domain, which may not be available at the stage of calibration in real-world applications and 2) their performance depends heavily on the disparity between the distributions of the source and target domains. To address these two limitations, we present novel calibration solutions via domain generalization. Our core idea is to leverage multiple calibration domains to reduce the effective distribution disparity between the target and calibration domains for improved calibration transfer without needing any data from the target domain. We provide theoretical justification and empirical experimental results to demonstrate the effectiveness of our proposed algorithms. Compared against state-of-the-art calibration methods designed for domain adaptation, we observe a decrease of 8.86 percentage points in expected calibration error or, equivalently, an increase of 35 percentage points in improvement ratio for multi-class classification on the Office-Home dataset.

A major impediment to the widespread deployment of intelligent training systems is the high cost of developing the content that drives their operation. Techniques grounded in end-user programming have shown great promise for reducing the burden of content creation. With these approaches, a domain expert demonstrates a solution to a task, which is then generalized to a broader model. This paper reports on a concept validation study that provides an empirical basis for the design of solution authoring frameworks based on end-user programming techniques. The study shows that non-expert users are comfortable with the approach and are capable of applying it to generate quality solution models. It also identifies constructs that, while important for accurate solution characterization, can lead to confusion and so warrant special care in tool design. Based on these results, we make recommendations for the design of solution-authoring tools in support of automated assessment for tutoring systems.

As computing devices become more pervasive in our daily lives, effective communication between the user and the system becomes increasingly important. The ability to describe actions at a human level of abstraction is key. However, the level at which computer system events is most easily captured is often well below the level at which humans conceptualize actions. We present a sequential pattern mining approach to discovering human-level actions-action idioms - from instrumentation logs of lower-level events. To support validation by a human expert, idiom discovery is designed to maximize recall, with filtering heuristics applied to help eliminate false positives. Empirical evaluation on data from a fielded application shows the promise of the approach for the automatic discovery of action idioms.

Learning by demonstration technology has long held the promise to empower non-programmers to customize and extend software. We describe the deployment of a learning by demonstration capability to support user creation of automated procedures in a collaborative planning environment that is used widely by the U.S. Army. This technology, which has been in operational use since the summer of 2010, has helped to reduce user workloads by automating repetitive and time-consuming tasks. The technology has also provided the unexpected benefit of enabling standardization of products and processes.

The Adept Task Learning system is an end-user programming environment that combines programming by demonstration and direct manipulation to support customization by nonprogrammers. Previously, Adept enforced a rigid procedure-authoring workflow consisting of demonstration followed by editing. However, a series of system evaluations with end users revealed a desire for more feedback during learning and more flexibility in authoring. We present a new approach that interleaves incremental learning from demonstration and assisted editing to provide users with a more flexible procedure-authoring experience. The approach relies on maintaining a "soup" of alternative hypotheses during learning, propagating user edits through the soup, and suggesting repairs as needed. We discuss the learning and reasoning techniques that support the new approach and identify the unique interaction design challenges they raise, concluding with an evaluation plan to resolve the design challenges and complete the improved system.

Intelligent systems require substantial bodies of problem-solving knowledge. Machine learning techniques hold much appeal for acquiring such knowledge but typically require extensive amounts of user-supplied training data. Alternatively, informed question asking can supplement machine learning by directly eliciting critical knowledge from a user. Question asking can reduce the amount of training data required, and hence the burden on the user; furthermore, focused question asking holds significant promise for faster and more accurate acquisition of knowledge. In previous work, we developed static strategies for question asking that provide background knowledge for a base learner, enabling the learner to make useful generalizations even with few training examples. Here, we extend that work with a learning approach for automatically acquiring question-asking strategies that better accommodate the interdependent nature of questions. We present experiments validating the approach and showing its usefulness for acquiring efficient, context-dependent question-asking strategies.

In a world of electronic calendars, the prospect of intelligent, personalized time management assistance seems a plausible and desirable application of AI. PTIME ( Personalized Time Management ) is a learning cognitive assistant agent that helps users handle email meeting requests, reserve venues, and schedule events. PTIME is designed to unobtrusively learn scheduling preferences, adapting to its user over time. The agent allows its user to flexibly express requirements for new meetings, as they would to an assistant. It interfaces with commercial enterprise calendaring platforms, and it operates seamlessly with users who do not have PTIME. This article overviews the system design and describes the models and technical advances required to satisfy the competing needs of preference modeling and elicitation, constraint reasoning, and machine learning. We further report on a multifaceted evaluation of the perceived usefulness of the system.

Recent years have seen a resurgence of interest in programming by demonstration. As end users have become increasingly sophisticated, computer and artificial intelligence technology has also matured, making it feasible for end users to teach long, complex procedures. This paper addresses the problem of learning from demonstrations involving unobservable (e.g., mental) actions. We explore the use of knowledge base inference to complete missing dataflow and investigate the approach in the context of the CALO cognitive personal desktop assistant. We experiment with the Pathfinder utility, which efficiently finds all the relationships between any two objects in the CALO knowledge base. Pathfinder often returns too many paths to present to the user and its default shortest path heuristic sometimes fails to identify the correct path. We develop a set of filtering techniques for narrowing down the results returned by Pathfinder and present experimental results showing that these techniques effectively reduce the alternative paths to a small, meaningful set suitable for presentation to a user.

Numerous techniques exist to help users automate repetitive tasks; however, none of these methods fully support end-user creation, use, and modification of the learned tasks. We present an integrated task learning system (ITL) that learns executable procedures based on user demonstration and instruction, constituting a first step toward a broader solution for procedure management. We discuss our deployment of ITL into a collaborative command-and-control system. In this complex domain, ITL's performance with end users doing real tasks indicates that providing multiple, integrated learning techniques both extends functionality and improves user experience. Our experience in integrat-ing this system also provides key insights for future designs of domain-independent task learning systems, specifically in supporting users' ability to understand and edit lengthy procedures.

We present PLIANT, a learning system that supports adaptive assistance in an open calendaring system. PLIANT learns user preferences from the feedback that naturally occurs during interactive scheduling. It contributes a novel application of active learning in a domain where the choice of candidate schedules to present to the user must balance usefulness to the learning module with immediate benefit to the user. Our experimental results provide evidence of PLIANT's ability to learn user preferences under various conditions and reveal the tradeoffs made by the different active learning selection strategies.

The Stock Tracker is an adaptive recommendation system for trading stocks that automatically acquires content-based models of user preferences to tailor its buy and sell advice. The system incorporates an efficient algorithm that exploits the fixed structure of user models and relies on unobtrusive data-gathering techniques. In this paper, we describe our approach to personalized recommendation and its implementation in this domain. We also discuss experiments that evaluate the system's behavior on both human subjects and synthetic users. The results suggest that the Stock Tracker can rapidly adapt its advice to different types of users

The Stock Tracker is a personalized recommendation system for trading stocks. The system tailors its buy, sell, and hold recommendations to individual users through automatically acquired content-based models of user preferences. It relies on data gathered unobtrusively during the natural course of interacting with a user.

The domain of crisis planning and scheduling taxes human response managers due to high levels of urgency and uncertainty.Such applications require assistant technologies (in contrast to automation technologies) and provide special challenges for interface design.We present INCA, the INteractive Crisis Assistant, that helps users develop effective crisis response plans and schedules in a timely manner.INCA also adapts to the individual users by anticipating their preferred responses to a given crisis and their intended repairs to a candidate response.We evaluate our system in HAZMAT, a synthetic domain involving hazardous material incidents.The results show that INCA provides effective support for the timely generation of effective responses and tailors itself to individual users.

The possibility of applying explanation-based learning (EBL), a technique from machine learning, to intelligent process planning is explored. There are currently two major approaches to process planning: the variant approach and the generative approach. Each has advantages and deficiencies. The authors' hypothesis is that EBL could successfully unite these apparently disparate approaches. EBL can be used to transform a traditional weak method planner into a strong method skeletal planner by acquiring strong method concepts which are generalized weak-method explanations of observed episodes. It would seem to be a natural vehicle to unite variant and generative process planning. A learning process planner, called EXBLIPP is implemented to test the authors' hypothesis. It is found that the system possesses many of the intended advantages. It is demonstrated that the EBL capability enables the process planning system to learn new schemata which yield many of the advantages of variant process planning.>