This paper introduces a testing strategy that is suitable for testing service-based applications. We describe an architecture that responds to changes of service operation, operation arguments and service composition changes. Our proof-of-concept test system performs runtime testing on our model atomic and composite web services using a random testing technique. A novel change identification method was developed to capture changes at the service interface. The test system is able to identify changes that occur in service operations and operational arguments in a service description of a test candidate. Our approach uses a new method to detect changes in a service inventory. Automated reconfiguration is used to support the continuous operation of the testing systems during a test candidate change.