A/B Testing is the gold standard to estimate the causal relationship between a change in a product and its impact on key outcome measures. It is widely used in the industry to test changes ranging from simple copy change or UI change to more complex changes like using machine learning models to personalize user experience. The key aspect of A/B testing is evaluation of experiment results. Designing the right set of metrics - correct outcome measures, data quality indicators, guardrails that prevent harm to business, and a comprehensive set of supporting metrics to understand the "why" behind the key movements is the #1 challenge practitioners face when trying to scale their experimentation program [11, 14]. On the technical side, improving sensitivity of experiment metrics is a hard problem and an active research area, with large practical implications as more and more small and medium size businesses are trying to adopt A/B testing and suffer from insufficient power. In this tutorial we will discuss challenges, best practices, and pitfalls in evaluating experiment results, focusing on both lessons learned and practical guidelines as well as open research questions. A version of this tutorial was also present at KDD 2019 [23]. It was attended by around 150 participants.

Traffic flow management (TFM) allocates the various airport, airspace, and other resources to maintain an efficient traffic flow consistent with safety. TFM is a complex area of research involving the disciplines of operations research, guidance and control, human factors, and software engineering. Hundreds of human operators make TFM decisions that involve tens of thousands of aircraft, en route air traffic control centers, the Federal Aviation Administration's System Command Center, and many airline operation centers. This paper provides an overview of how TFM decisions are made today and challenges facing the system in the future, and reviews modeling and optimization approaches for facilitating system-wide modeling, performance assessments, and system-level optimization of the national airspace system in the presence of both en route and airport capacity constraints.