As chatbots driven by large language models (LLMs) are increasingly deployed in everyday contexts, their ability to recover from errors through effective apologies is critical to maintaining user trust and satisfaction. In a preregistered study with Prolific workers ( N = 162), we examine user preferences for three types of apologies ( rote , explanatory , and empathic ) issued in response to three categories of common LLM mistakes ( bias , unfounded fabrication , and factual errors ). We designed a pairwise experiment in which participants evaluated chatbot responses consisting of an initial error, a subsequent apology, and a resolution. Explanatory apologies were generally preferred, but this varied by context and user. In the bias scenario, empathic apologies were favored for acknowledging emotional impact, while hallucinations, though seen as serious, elicited no clear preference, reflecting user uncertainty. Our findings show the complexity of effective apology in AI systems. We discuss key insights such as personalization and calibration that future systems must navigate to meaningfully repair trust.

As artificial intelligence and machine learning algorithms become increasingly prevalent in society, multiple stakeholders are calling for these algorithms to provide explanations. At the same time, these stakeholders, whether they be affected citizens, government regulators, domain experts, or system developers, have different explanation needs. To address these needs, in 2019, we created AI Explainability 360, an open source software toolkit featuring ten diverse and state-of-the-art explainability methods and two evaluation metrics. This paper examines the impact of the toolkit with several case studies, statistics, and community feedback. The different ways in which users have experienced AI Explainability 360 have resulted in multiple types of impact and improvements in multiple metrics, highlighted by the adoption of the toolkit by the independent LF AI & Data Foundation. The paper also describes the flexible design of the toolkit, examples of its use, and the significant educational material and documentation available to its users.

Generative machine learning models have recently been applied to source code, for use cases including translating code between programming languages, creating documentation from code, and auto-completing methods. Yet, state-of-the-art models often produce code that is erroneous or incomplete. In a controlled study with 32 software engineers, we examined whether such imperfect outputs are helpful in the context of Java-to-Python code translation. When aided by the outputs of a code translation model, participants produced code with fewer errors than when working alone. We also examined how the quality and quantity of AI translations affected the work process and quality of outcomes, and observed that providing multiple translations had a larger impact on the translation process than varying the quality of provided translations. Our results tell a complex, nuanced story about the benefits of generative code models and the challenges software engineers face when working with their outputs. Our work motivates the need for intelligent user interfaces that help software engineers effectively work with generative code models in order to understand and evaluate their outputs and achieve superior outcomes to working alone.

Generative models have become adept at producing artifacts such as images, videos, and prose at human-like levels of proficiency. New generative techniques, such as unsupervised neural machine translation (NMT), have recently been applied to the task of generating source code, translating it from one programming language to another. The artifacts produced in this way may contain imperfections, such as compilation or logical errors. We examine the extent to which software engineers would tolerate such imperfections and explore ways to aid the detection and correction of those errors. Using a design scenario approach, we interviewed 11 software engineers to understand their reactions to the use of an NMT model in the context of application modernization, focusing on the task of translating source code from one language to another. Our three-stage scenario sparked discussions about the utility and desirability of working with an imperfect AI system, how acceptance of that system’s outputs would be established, and future opportunities for generative AI in application modernization. Our study highlights how UI features such as confidence highlighting and alternate translations help software engineers work with and better understand generative NMT models.

As artificial intelligence algorithms make further inroads in high-stakes societal applications, there are increasing calls from multiple stakeholders for these algorithms to explain their outputs. To make matters more challenging, different personas of consumers of explanations have different requirements for explanations. Toward addressing these needs, we introduce AI Explainability 360, an open-source Python toolkit featuring ten diverse and state-of-the-art explainability methods and two evaluation metrics. Equally important, we provide a taxonomy to help entities requiring explanations to navigate the space of interpretation and explanation methods, not only those in the toolkit but also in the broader literature on explainability. For data scientists and other users of the toolkit, we have implemented an extensible software architecture that organizes methods according to their place in the AI modeling pipeline. The toolkit is not only the software, but also guidance material, tutorials, and an interactive web demo to introduce AI explainability to different audiences. Together, our toolkit and taxonomy can help identify gaps where more explainability methods are needed and provide a platform to incorporate them as they are developed.

Tommy Sandbank, Michal Shmueli-Scheuer, Jonathan Herzig, David Konopnicki, John Richards, David Piorkowski. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). 2018.

Over 100 top-traffic and government websites from the United States and United Kingdom were examined for evidence of changes on accessibility indicators over the 14-year period from 1999 to 2012, the longest period studied to date. Automated analyses of WCAG 2.0 Level A Success Criteria found high percentages of violations overall. Unlike more circumscribed studies, however, these sites exhibited improvements over the years on a number of accessibility indicators, with government sites being less likely than topsites to have accessibility violations. Examination of the causes of success and failure suggests that improving accessibility may be due, in part, to changes in website technologies and coding practices rather than a focus on accessibility per se.

This paper explores evidence for the conjecture that improvements in Web accessibility have arisen, in part, as side effects of changes in Web technology and associated shifts in the way Web pages are designed and coded. Drawing on an earlier study of Web accessibility trends over the past 14 years, it discusses several possible indirect contributors to improving accessibility including the use of new browser capabilities to create more sophisticated page layouts, a growing concern with improved page rank in search results, and a shift toward cross-device content design. Understanding these examples may inspire the creation of additional technologies with incidental accessibility benefits.

In this study, we explore the role of age and fluid intelligence on the behavior of people looking for information in a real-world search space. Analyses of mouse moves, clicks, and eye movements provide a window into possible differences in both task strategy and performance, and allow us to begin to separate the influence of age from the correlated but isolable influence of cognitive ability. We found little evidence of differences in strategy between younger and older participants matched on fluid intelligence. Both performance and strategy differences were found between older participants having higher versus lower fluid intelligence, however, suggesting that cognitive factors, rather than age per se, exert the dominant influence. This underscores the importance of measuring and controlling for cognitive abilities in studies involving older adults.

Sketching facilitates design in the exploration of ideas about concrete objects and abstractions. In fact, throughout the software engineering process when grappling with new ideas, people reach for a pen and start sketching. While pen and paper work well, digital media can provide additional features to benefit the sketcher. Digital support will only be successful, however, if it does not detract from the core sketching experience. Based on research that defines characteristics of sketches and sketching, this paper offers three preliminary tool examples. Each example is intended to enable sketching while maintaining its characteristic experience.

Designers often have no access to individuals who use screen reading software, and may have little understanding of how their design choices impact these users. We explore here whether cog-nitive models of auditory interaction could provide insight into screen reader usability. By comparing human data with a tool-generated model of a practiced task performed using a screen reader, we identify several requirements for such models and tools. Most important is the need to represent parallel execution of hearing with thinking and acting. Rules for placement of cogni-tive operators that were developed for visual user interfaces may not be applicable in the auditory domain. Other mismatches be-tween the data and the model were attributed to the extremely fast listening rate and differences between the typing patterns of screen reader usage and the model's assumptions. This work in-forms the development of more accurate models of auditory inter-action. Tools incorporating such models could help designers create user interfaces that are well tuned for screen reader users, without the need for modeling expertise.

Over the last few years, we have built and tested two systems designed to make Web content more accessible for people with limited vision and dexterity. The first system, based on content transcoding via a proxy server, possessed several attractive features but proved to be unacceptably complex, error prone, and slow. The second system, based on client-side transformations, worked well enough to be broadly deployed. We report here on lessons learned and on the current state of the research effort. We review the two systems, discuss their strengths and weaknesses, and examine how the second system is being used.

We report here on our progress on a project first described at the ASSETS 2002 conference. At that time, we had developed a prototype system in which a proxy server intermediary was used to adapt Web pages to meet the needs of older adults. Since that report, we field tested the prototype and learned of problems with the proxy approach. We report on the lessons learned from that work and on our new approach towards meeting the Web needs of older adults and users with disabilities. This new software makes adaptations on the client machine, with greater accuracy and speed than was possible with the proxy server approach. It transforms Web pages "on the fly", without requiring that all Web content be re-written. The new software has been in use for a year and we report here on our findings from the usage. We discuss this approach in the context of Web accessibility standards and Web usability.

Web accessibility is an important goal. However, most approaches to its attainment are based on unrealistic economic models in which Web content developers are required to spend too much for which they receive too little. We believe this situation is due, in part, to the overly narrow definitions given both to those who stand to benefit from enhanced access to the Web and what is meant by this enhanced access. In this paper, we take a broader view, discussing a complementary approach that costs developers less and provides greater advantages to a larger community of users. While we have quite specific aims in our technical work, we hope it can also serve as an example of how the technical conversation regarding Web accessibility can move beyond the narrow confines of limited adaptations for small populations.

We take as our premise that it is possible and desirable to design systems that support social processes. We describe Loops, a project which takes this approach to supporting computer-mediated communication (CMC) through structural and intemctive properties such as persistence and a minimalist graphical representation of users and their activities that we call a social proxy. We discuss a prototype called Babble that has been used by our group for over a year, and has been deployed to six other groups at the Watson labs for about two months. We describe usage experiences, lessons learned, and next steps.

No abstract available.

This paper discusses a set of tools supporting the rapid development of voice and telephony applications. The tools allow interfaces to be rapidly prototyped, tested and installed without impacting the underlying system. Used directly by behavioral specialists, they have played a key role in the building of two production systems. We review several essential features of this facility and then outline its role in the rapid development of a voice messaging system for the athletes and officials at the 1984 Summer Olympics in Los Angeles.