Boneh, Bonneau, Bünz, and Fisch (CRYPTO 2018) recently introduced the notion of a verifiable delay function (VDF). VDFs are functions that take a long sequential time T to compute, but whose outputs y := Eval(x) can be efficiently verified (possibly given a proof π) in time t ≪ T (e.g., t = poly(λ, log T) where λ is the security parameter). The first security requirement on a VDF, called uniqueness, is that no polynomial-time algorithm can find a convincing proof π' that verifies for an input x and a different output y' ≠ y. The second security requirement, called sequentiality, is that no polynomial-time algorithm running in time σ < T for some parameter σ (e.g., σ = T^{1/10}) can compute y, even with poly(T,λ) many parallel processors. Starting from the work of Boneh et al., there are now multiple constructions of VDFs from various algebraic assumptions. In this work, we study whether VDFs can be constructed from ideal hash functions in a black-box way, as modeled in the random oracle model (ROM). In the ROM, we measure the running time by the number of oracle queries and the sequentiality by the number of rounds of oracle queries. We rule out two classes of constructions of VDFs in the ROM: - We show that VDFs satisfying perfect uniqueness (i.e., VDFs where no different convincing solution y' ≠ y exists) cannot be constructed in the ROM. More formally, we give an attacker that finds the solution y in ≈ t rounds of queries, asking only poly(T) queries in total. - We also rule out tight verifiable delay functions in the ROM. Tight verifiable delay functions, recently studied by Döttling, Garg, Malavolta, and Vasudevan (ePrint Report 2019), require sequentiality for σ ≈ T-T^ρ for some constant 0 < ρ < 1. More generally, our lower bound also applies to proofs of sequential work (i.e., VDFs without the uniqueness property), even in the private verification setting, and sequentiality σ > T-(T)/(2t) for a concrete verification time t.