Low reliability in Web-based applications can result in detrimental effects for business, government, and consumers as they become increasingly dependent on the Internet for routine operations. A short time to market, large user community, demand for continuous availability, and frequent updates motivate automated, cost-effective testing strategies. To investigate the practical tradeoffs of different automated strategies for key components of the Web-based software testing process, we have designed a framework for Web-based software testing that focuses on scalability and evolving the test suite automatically as the application's operational profile changes. We have developed an initial prototype that not only demonstrates how existing tools can be used together but provides insight into the cost effectiveness of the overall approach. This paper describes the testing framework, discusses the issues in building and reusing tools in an integrated manner, and presents a case study that exemplifies the usability, costs, and scalability of the approach.