Regulatory agencies often operate with limited resources and rely on tips from the public to identify potential violations.However, processing these tips at scale presents significant operational challenges, as agencies must correctly identify and route relevant tips to the appropriate enforcement divisions.Through a case study, we demonstrate how advances in large language models can be utilized to support overburdened agencies with limited capacities.In partnership with the U.S. Environmental Protection Agency, we leverage previously unstudied citizen tips data from their "Report a Violation" system to develop an LLM-assisted pipeline for tip routing.Our approach filters out 80.5% of irrelevant tips and increases overall routing accuracy from 31.8% to 82.4% compared to the current routing system.At a time of increased focus on government efficiencies, our approach provides a constructive path forward by using technology to empower civil servants. * Equal contribution.

Most existing evaluations of explainable machine learning (ML) methods rely on simplifying assumptions or proxies that do not reflect real-world use cases; the handful of more robust evaluations on real-world settings have shortcomings in their design, generally leading to overestimation of methods' real-world utility. In this work, we seek to address this by conducting a study that evaluates post-hoc explainable ML methods in a setting consistent with the application context and provide a template for future evaluation studies. We modify and improve a prior study on e-commerce fraud detection by relaxing the original work's simplifying assumptions that departed from the deployment context. Our study finds no evidence for the utility of the tested explainable ML methods in the context, which is a drastically different conclusion from the earlier work. This highlights how seemingly trivial experimental design choices can yield misleading conclusions about method utility. In addition, our work carries lessons about the necessity of not only evaluating explainable ML methods using tasks, data, users, and metrics grounded in the intended application context but also developing methods tailored to specific applications, moving beyond general-purpose explainable ML methods.

Rental assistance programs provide individuals with financial assistance to prevent housing instabilities caused by evictions and avert homelessness. Since these programs operate under resource constraints, they must decide who to prioritize. Typically, funding is distributed by a reactive allocation process that does not systematically consider risk of future homelessness. We partnered with Anonymous County (PA) to explore a proactive and preventative allocation approach that prioritizes individuals facing eviction based on their risk of future homelessness. Our ML models, trained on state and county administrative data accurately identify at-risk individuals, outperforming simpler prioritization approaches by at least 20% while meeting our equity and fairness goals across race and gender. Furthermore, our approach would reach 28% of individuals who are overlooked by the current process and end up homeless. Beyond improvements to the rental assistance program in Anonymous County, this study can inform the development of evidence-based decision support tools in similar contexts, including lessons about data needs, model design, evaluation, and field validation.

As data science and machine learning (ML) increasingly shape our society, the importance of developing fair algorithmic decision-making systems becomes paramount. There is a pressing need to train data scientists and practitioners on handling bias and fairness in real-world scenarios, from early stages of a data science project to maintaining ML systems in production. Existing resources are mostly academic and cover the ML training and optimization aspects of bias mitigation, leaving practitioners without comprehensive frameworks for making decisions throughout a real-world project lifecycle. This tutorial aims to bridge the gap between research and practice, providing an in-depth exploration of algorithmic fairness, encompassing metrics and definitions, practical case studies, data bias understanding, bias mitigation and model fairness audits using the Aequitas toolkit. Participants will be equipped to engage in conversations about bias, assist decision-makers in understanding options and trade-offs, evaluate project scoping aspects influencing fairness outcomes, and define actions and interventions based on model predictions. They will also learn to identify cohorts, target variables, evaluation metrics, and establish bias and fairness goals for different groups. Moreover, participants will gain insights into auditing and mitigating model bias, and implementing continuous monitoring to assess retraining needs. The tutorial addresses the current lack of practical training materials, methodologies, and tools for researchers and developers working on real-world algorithmic decision-making systems. By the conclusion of this hands-on tutorial, attendees will be well-versed in navigating bias-related issues, selecting appropriate metrics, and applying bias audit and mitigation frameworks and tools for informed design decisions in real-world data science systems.

Tackling issues of bias and fairness when building and deploying data science systems has received increased attention from the research community in recent years, yet a lot of the research has focused on theoretical aspects and very limited set of application areas and data sets. There is a lack of 1) practical training materials, 2) methodologies, and 3) tools for researchers and developers working on real-world algorithmic decision making system to deal with issues of bias and fairness. Today, treating bias and fairness as primary metrics of interest, and building, selecting, and validating models using those metrics is not standard practice for data scientists. In this hands-on tutorial we will try to bridge the gap between research and practice, by deep diving into algorithmic fairness, from metrics and definitions to practical case studies, including bias audits using the Aequitas toolkit (http://github.com/dssg/aequitas). By the end of this hands-on tutorial, the audience will be familiar with bias mitigation frameworks and tools to help them making decisions during a project based on intervention and deployment contexts in which their system will be used.