Cyber-Physical Systems (CPSs) are composed of computational control logic and physical processes, which intertwine with each other. CPSs are widely used in various domains of daily life, including those safety-critical systems and infrastructures, such as medical monitoring, autonomous vehicles, and water treatment systems. It is thus critical to effectively test them. However, it is not easy to obtain test cases which can fail the CPS. In this work, we propose a failure-inducing input generation approach FIGCPS, which requires no knowledge of the CPS under test or any history logs of the CPS which are usually hard to obtain. Our approach adopts deep reinforcement learning techniques to interact with the CPS under test and effectively searches for failure-inducing input guided by rewards. Our approach adaptively collects information from the CPS, which reduces the training time and is also able to explore different states. Moreover, our approach is the first attempt to generate failure-inducing input for CPSs with both continuous action space and high-dimensional discrete action space, which are common for some classes of CPSs. The evaluation results show that FIGCPS not only achieves a higher success rate than the state-of-the-art approaches but also finds two new attacks in a well-tested CPS.