As machine learning (ML) becomes increasingly popular, developers without deep experience in ML - who we will refer to as ML practitioners - are facing the need to diagnose problems with ML models. Yet successful diagnosis requires high-level expertise that practitioners lack. As in many complex data-oriented domains, visualization could help. This two-phase study explored the design of visualizations to aid ML diagnosis. In phase 1, twelve ML practitioners were asked to diagnose a model using ten state-of-the-art visualizations; seven design themes were identified. In phase 2, several design themes were embodied in an interactive visualization. The visualization was used to engage practitioners in a participatory design exercise that explored how they would carry out multi-step diagnosis using the visualization. Our findings provide design implications for tools that better support ML diagnosis by non-expert practitioners.