This paper introduces the concept of Modular Hydraulic Propulsion, in which a modular robot that operates in a fluid environment moves by routing the fluid through itself. The robot's modules represent sections of a hydraulics network. Each module can move fluid between any of its faces. The modules (network sections) can be rearranged into arbitrary topologies. We propose a decentralized motion controller, which does not require modules to communicate, compute, nor store information during run-time. We use 3-D simulations to compare the performance of this controller to that of a centralized controller with full knowledge of the task. We also detail the design and fabrication of six 2-D prototype modules, which float in a water tank. Results of systematic experiments show that the decentralized controller, despite its simplicity, reliably steers modular robots towards a light source. Modular Hydraulic Propulsion could offer new solutions to problems requiring reconfigurable systems to move precisely in 3-D, such as inspection of pipes, vascular systems or other confined spaces.