Computer-Aided Design (CAD) sketches, composed of geometric primitives and constraints, are fundamental to CAD models and play a critical role in industrial design and manufacturing. Leveraging artificial intelligence to convert hand-drawn sketches and rendered images into CAD sketches has the potential to streamline and accelerate the design process. Existing approaches predominantly focus on separately learning primitives and constraints, often employing two-stage methods or learnable tokens to model these elements independently. However, such methods fail to fully exploit the intrinsic relationships between primitives and constraints. In this paper, we propose LuBan, a lightweight, end-to-end model for CAD sketch generation that eliminates the need for separate constraint models or tokens. LuBan leverages the DEtection TRansformer (DETR) architecture for primitive modeling and distinguishes between parametric and non-parametric features. By deriving sub-primitive features from the intermediate outputs of the primitive model, LuBan facilitates constraint prediction, effectively capturing the inherent relationships between primitives and constraints. This enables the generation of CAD sketches as hierarchical graphs. Qualitative and quantitative experiments on both precise and hand-drawn renderings demonstrate that LuBan achieves state-of-the-art performance. Ablation studies further confirm its superiority over independently trained primitive models, validating its effectiveness. Additionally, LuBan embodies the principle of "what you draw is what you get," offering significant enhancements to the design process for designers.