Violating the Von Neumann sequential processing principle at the microarchitectural level is commonplace to reach high performing CPU hardware — violations are safe as long as software executes correctly at the architectural interface. Speculative execution attacks exploit these violations and queue up secret-dependent memory accesses allowed by long speculation windows due to the late detection of these violations in the pipeline. In this paper, we show that recent AMD and Intel CPUs speculate very early in their pipeline, even before they decode the current instruction. This mechanism enables new sources of speculation to be triggered from almost any instruction, enabling a new class of attacks that we refer to as Phantom. Unlike Spectre, Phantom speculation windows are short since the violations are detected early. Nonetheless, Phantom allows for transient fetch and transient decode on all recent x86-based microarchitectures, and transient execution on AMD Zen 1 and 2. We build a number of exploits using these new Phantom primitives and discuss why mitigating them is difficult in practice.