The integration of Large Language Models (LLMs) into recommender systems has introduced a powerful paradigm for personalized recommedation. However, due to biases stemming from sensitive user attributes (e.g., gender, age), LLMs may generate discriminatory and unfair recommendations, even for users with similar preferences. Existing methods to improve LLM-based recommendation fairness, like prompt engineering or data-processing, are often unstable and fail to address unfairness rooted in pre-training data and model parameters. While fine-tuning approaches show promise, improving fairness typically comes at the cost of degraded recommendation performance, as the model's parameters conflates user preferences with sensitive attributes. To address this, we propose FairSpec, a novel lightweight fine-tuning framework designed to enhance fairness while preserving recommendation quality. FairSpec has a Parameter-Efficient Mixture-of-Experts architecture (PE-MoE) with a fairness-aware router. The experts are specialized into utility experts and fairness experts, and the latter are adversarially trained to suppress sensitive information. Furthermore, FairSpec introduces Fairness-Aware Direct Preference Optimization (FA-DPO), which employs a composite reward function to jointly optimize recommendation performance and fairness. Results on three public benchmarks with four LLM backbones demonstrate that FairSpec achieves the best overall performance-fairness balance, attaining the top average rank across all metrics. Furthermore, visualizations of routing weights illustrate the strategic behavior and effectiveness of the proposed fairness-aware routing mechanism. Our code can be found at https://github.com/qin2mu4/FairSpec.