State-of-the-art exoskeletons are typically limited by low control bandwidth and small range stiffness of actuators which are based on high gear ratios and elastic components (e.g., series elastic actuators). Furthermore, most exoskeletons are based on discrete gait phase detection and/or discrete stiffness control resulting in discontinuous torque profiles. To fill these two gaps, we developed a portable lightweight knee exoskeleton using quasi-direct drive (QDD) actuation that provides 14 Nm torque (36.8% biological joint moment for overground walking). This paper presents 1) stiffness modeling of torque-controlled QDD exoskeletons and 2) stiffness-based continuous torque controller that estimates knee joint moment in real-time. Experimental tests found the exoskeleton had high bandwidth of stiffness control (16 Hz under 100 Nm/rad) and high torque tracking accuracy with 0.34 Nm Root Mean Square (RMS) error (6.22%) across 0-350 Nm/rad large range stiffness. The continuous controller was able to estimate knee moments accurately and smoothly for three walking speeds and their transitions. Experimental results with 8 able-bodied subjects demonstrated that our exoskeleton was able to reduce the muscle activities of all 8 measured knee and ankle muscles by 8.60%-15.22% relative to unpowered condition, and two knee flexors and one ankle plantar flexor by 1.92%-10.24% relative to baseline (no exoskeleton) condition.