We propose a two-layer control architecture for bilateral teleoperation with communication delays. The controller is structured with an (inner) performance layer and an (outer) passivity layer. In the performance layer, any traditional controller for bilateral teleoperation can be implemented. The passivity layer guarantees that, from the operator and environment perspective, the overall teleoperator is passive: The amount of energy that can be extracted from the teleoperator is bounded from below and the rate of increase of the stored energy in the teleoperator is bounded by (twice) the environment and operator supplied power. Passivity is ensured by modulating the performance layer outputs and by injecting a variable amount of damping via an energy-based logic that follows the innovative principle of energy duplication and takes into account the detrimental effects of time delays. In contrast to the traditional teleoperation approach, in which the master and slave controllers implement an as-stiff-as-possible coupling between the master and slave devices, our scheme is specifically designed for direct force-reflecting bilateral teleoperation: The slave controller mimics the operator action, whereas the master controller reflects the slave-environment interaction. We illustrate the performance of the two-layer approach in a challenging experiment with a round-trip communication delay of 300 ms while making and breaking contact with a stiff aluminum environment. Finally, we also compare our controller with the state of the art.