Wireless cellular networks have many parameters that are normally tuned upon deployment and re-tuned as the network changes. Many operational parameters affect reference signal received power (RSRP), reference signal received quality (RSRQ), signal-to-interference-plus-noise-ratio (SINR), and, ultimately, throughput. In this paper, we develop and compare two approaches for maximizing coverage and minimizing interference by jointly optimizing the transmit power and downtilt (elevation tilt) settings across sectors. To evaluate different parameter configurations offline, we construct a realistic simulation model that captures geographic correlations. Using this model, we evaluate two optimization methods: deep deterministic policy gradient (DDPG), a reinforcement learning (RL) algorithm, and multi-objective Bayesian optimization (BO). Our simulations show that both approaches significantly outperform random search and converge to comparable Pareto frontiers, but that BO converges with two orders of magnitude fewer evaluations than DDPG. Our results suggest that data-driven techniques can effectively self-optimize coverage and capacity in cellular networks.