Accurate lithology identification from well log data is fundamental for reservoir characterization, yet its practical application is often impeded by the dual challenges of incomplete well log data and scarce labeled samples. Existing data-driven methods often address these issues separately or require extensive complete labeled data, limiting their effectiveness in low-resource geological exploration scenarios. To address this compound challenge, this paper proposes a multi-constraint guided self-supervised diffusion framework, designated as MCG-SSDF, based on a generative pre-training paradigm. The framework first employs a novel multi-constraint guided diffusion model, MCG-Diff, for unsupervised pre-training on large-scale unlabeled data. This model integrates a Mamba backbone with three geological priors, namely a global structural constraint, a petrophysical correlation constraint, and a geological morphological constraint, to enhance the fidelity and physical plausibility of well log data generation and imputation. Subsequently, a parameter-efficient fine-tuning strategy is applied, where only a lightweight classification head is trained on a small set of labeled, incomplete samples, enabling the framework to perform end-to-end lithology identification directly from incomplete data during the inference stage. Systematic evaluations on two oilfield datasets validate the framework across deterministic accuracy and uncertainty quantification. Results reveal that MCG-SSDF outperforms baselines, particularly under extreme label scarcity. Ablation studies and attribution analyses further confirm the synergistic contributions and decision reliability of integrated geological constraints. This methodology provides a robust solution for high-precision lithology identification in low-resource environments.