We describe a usability inspection method to evaluate Application Programming Interfaces (APIs). We found the method useful as it identified usability defects in Microsoft's .NET Framework, of which 59% were new and 21% were fixed. Based on a comparison of usability defects identified between API usability peer reviews and API usability tests, API usability tests were found to expose design issues related to actually using an API whereas API usability peer reviews were found to expose the design rationale of an API. We reflect on the efficiency and productivity of each method: each API usability test is equivalent to 16 API usability peer reviews with the former having a 2.5x productivity advantage. We discuss how API usability peer reviews can be used in conjunction with API usability tests to increase usability coverage on APIs.

API usability tests in the lab are time and resource intensive, thus allowing a relatively small percentage of the API namespace to be evaluated. We describe a group-based usability inspection method -- API Peer Reviews -- to evaluate API usability. Based on an analysis of usability breakdowns from API Peer Reviews and API usability tests, results show that API Peer Reviews identified breakdowns across several cognitive dimensions, some of which were different than what was identified by API usability tests. We reflect on the adoption of API Peer Reviews as a collaborative practice in organizations for evaluating API usability.

Usability testing is a relatively new and rapidly developing field. Newcomers to usability testing typically enter the profession with a knowledge of social science research methods and the belief that usability testing is conducted in a lab using thinking aloud techniques to identify usability defects. Our practice of usability testing at Mead Data Central has shown that these core beliefs represent an approach of limited utility in designing products like the LEX1S®NEXIS® research systems. In this paper, we describe our experiences in conducting traditional usability testing and how we used the results of those efforts to develop more effective methods of testing for Mead Data Central's products and customers.