Neural networks attract significant attention in almost every field due to their widespread applications in various tasks. However, developers often struggle with debugging due to the black-box nature of neural networks. Visual analytics provides an intuitive way for developers to understand the hidden states and underlying complex transformations in neural networks. Existing visual analytics tools for neural networks have been demonstrated to be effective in providing useful hints for debugging certain network architectures. However, these approaches are often architecture-specific with strong assumptions of how the network should be understood. This limits their use when the network architecture or the exploration goal changes. In this paper, we present a general model and a programming toolkit, Neural Network Visualization Builder (NNVisBuilder), for prototyping visual analytics systems to understand neural networks. NNVisBuilder covers the common data transformation and interaction model involved in existing tools for exploring neural networks. It enables developers to customize a visual analytics interface for answering their specific questions about networks. NNVisBuilder is compatible with PyTorch so that developers can integrate the visualization code into their learning code seamlessly. We demonstrate the applicability by reproducing several existing visual analytics systems for networks with NNVisBuilder. The source code and some example cases can be found at https://github.com/sysuvis/NVB.