Testing security systems is challenging because a system's authors have to play the double role of attackers and defenders. Red team/blue team exercises are an invaluable mechanism for security testing. They partition researchers into two competing teams of attackers and defenders, enabling them to create challenging and realistic test scenarios. While such exercises provide valuable insight into vulnerabilities of security systems, they are very expensive and thus rarely performed. In this paper we describe a red team/blue team exercise, sponsored by DARPA's FTN program, and performed October 2002 --- May 2003. The goal of the exercise was to evaluate a collaborative DDoS defense, comprised of a distributed system, COSSACK, and a stand-alone defense, D-WARD. The role of the blue team was played by developers of the tested systems from USC/ISI and UCLA, the red team included researchers from Sandia National Laboratory, and all the coordination, experiment execution, result collection and analysis was performed by the white team from BBN Technologies. This exercise was of immense value to all involved --- it uncovered significant vulnerabilities in tested systems, pointed out desirable characteristics in DDoS defense systems (e.g., avoiding reliance on timing mechanisms), and taught us many lessons about testing of DDoS defenses.