We propose and analyze a new policy gradient algorithm for reinforcement learning (RL), relative Monte Carlo (rMC). The method estimates policy gradients using relative returns between a root sample path and counterfactual simulated paths, instantiated by taking a different action from the root. The resulting gradient estimate is both unbiased and has low variance. rMC is compatible with any differentiable policy, including neural networks, and is guaranteed to converge even for infinite horizon tasks. The method utilizes common random number coupling of the simulated paths to reduce variance and increase the likelihood that paths merge, thereby reducing simulation complexity. It is particularly well suited to discrete event control problems where actions have a "local" effect, such as queueing, supply chain, or ride-hailing problems. Indeed, we show that it has provably low complexity for a family of inventory control problems. Numerical tests on a challenging inventory and fulfillment problem show that compared to traditional RL approaches, rMC converges in far fewer iterations (lower variance), has better policy performance (unbiased), and requires minimal hyperparameter tuning.