Audits are critical mechanisms for identifying the risks and limitations of deployed artifcial intelligence (AI) systems.However, the efective execution of AI audits remains incredibly difcult, and practitioners often need to make use of various tools to support their eforts.Drawing on interviews with 35 AI audit practitioners and a landscape analysis of 435 tools, we compare the current ecosystem of AI audit tooling to practitioner needs.While many tools are designed to help set standards and evaluate AI systems, they often fall short in supporting accountability.We outline challenges practitioners faced in their eforts to use AI audit tools and highlight areas for future tool development beyond evaluationfrom harms discovery to advocacy.We conclude that the available resources do not currently support the full scope of AI audit practitioners' needs and recommend that the feld move beyond tools for just evaluation and towards more comprehensive infrastructure for AI accountability.

A few large, homogenous, pre-trained models undergird many machine learning systems -and often, these models contain harmful stereotypes learned from the internet.We investigate the bias transfer hypothesis: the theory that social biases (such as stereotypes) internalized by large language models during pre-training transfer into harmful task-specific behavior after fine-tuning.For two classification tasks, we find that reducing intrinsic bias with controlled interventions before finetuning does little to mitigate the classifier's discriminatory behavior after fine-tuning.Regression analysis suggests that downstream disparities are better explained by biases in the fine-tuning dataset.Still, pre-training plays a role: simple alterations to co-occurrence rates in the fine-tuning dataset are ineffective when the model has been pre-trained.Our results encourage practitioners to focus more on dataset quality and context-specific harms.