A light field (LF), which is represented as a set of dense multiview images, has been utilized in various 3-D applications. To make LF acquisition more efficient, researchers have investigated compressive sensing methods by incorporating modulation or coding functions into the camera. In this work, we investigate a challenging case of compressive LF acquisition in which an entire LF should be reconstructed from only a single coded image. To achieve this goal, we propose a new modulation scheme called factorized modulation that can approximate arbitrary 4-D modulation patterns in a factorized manner. Our method can be hardware-implemented by combining the architectures for coded aperture and pixel-wise coded exposure imaging. The modulation pattern is jointly optimized with a CNN-based reconstruction algorithm. Our method is validated through extensive evaluations against other modulation schemes.