Commercial content moderation APIs are marketed as scalable solutions to combat online hate speech. However, the reliance on these APIs risks both silencing legitimate speech, called over-moderation, and failing to protect online platforms from harmful speech, known as under-moderation. To assess such risks, this paper introduces a framework for auditing black-box NLP systems. Using the framework, we systematically evaluate five widely used commercial content moderation APIs. Analyzing five million queries based on four datasets, we find that APIs frequently rely on group identity terms, such as “black”, to predict hate speech. While OpenAI’s and Amazon’s services perform slightly better, all providers under-moderate implicit hate speech, which uses codified messages, especially against LGBTQIA+ individuals. Simultaneously, they over-moderate counter-speech, reclaimed slurs and content related to Black, LGBTQIA+, Jewish, and Muslim people. We recommend that API providers offer better guidance on API implementation and threshold setting and more transparency on their APIs’ limitations.Warning: This paper contains offensive and hateful terms and concepts. We have chosen to reproduce these terms for reasons of transparency.

Cloud services for natural language processing (NLP) increasingly establish as viable alternatives to self-maintained and self-trained NLP pipelines. In particular, they feature low access barriers and management overhead, a pay-as-you-go pricing model, and elastic scalability allowing to process large amounts of natural language data ad hoc. Any deliberation about employing cloud NLP services in practice does, however, face the challenge that so far, little is known about the accuracy provided by such services as well as about how to conduct respective quality assessments.In this paper, we therefore present a method for evaluating the accuracy provided by cloud NLP services and apply it to cloud services for three prominent NLP tasks offered by Amazon, Google, Microsoft, and IBM. Our results show significantly different accuracies as well as different dependencies on the specifics of input data among the covered providers. Our insights therefore allow for a more evidence-based quality-driven choice of the provider to be used for NLP in practice. Furthermore, the general approach employed may also serve as a blueprint for additional future evaluations of cloud NLP services for other tasks or offered by other providers.