In this paper, we report our experience testing the Gener-icModelAgent toolkit, a critical piece of the software developed for the NSF CyberWater project [1], to ensure that the workflows work properly without error when deployed to the user community.The many challenges we faced include a lack of developer-oriented specifications, a complex framework, complicated and domain-specific functionality, frequently changing requirements, domain developers' unfamiliarity with TDD best practices, and poor model-view separation in a legacy software used to construct visual workflows.We present a generalized testing strategy for approaching different modules in this toolkit, in an effort to understand every input, its relation to the main functionality of the module, and every output.We then built unit and integration test cases from bottom up to enable systematic and thorough exercises of all usage scenarios identified of every module as well as their integration.Our testing discovered problems in the toolkit with suggested fixes that no one was aware of before.It also led to the creation of rewritten specifications for each module, as a by-product, that are precise and developer-and tester-(rather than user-) oriented.Our specifications, test plan, test cases and results became reusable assets for future development, testing and maintenance of the toolkit.

Composed of developers with diverse backgrounds in multiple disciplines, the NSF CyberWater project team needed to research and implement effective software testing methods to improve the team's workflow efficiency and software quality.In this paper we present a practical and effective strategy for automated black-box testing of Cyber-Water modules using a Continuous Integration / Continuous Deployment (CI/CD) pipeline and the Jenkins automation server, Python unittest and ptest, and a novel technique we call object-method replacement, which isolates the backend from the front-end logic.Our experience can be adapted and extended to other research projects to mitigate the risk of programming errors and mistakes incurred through continuous development on a code repository.