Despite the recent success of large language models (LLMs) in time-series forecasting, most existing methods still adopt a Deep Synchronous Fusion strategy, where dense interactions between textual and temporal features are enforced at every layer of the network.This design overlooks the inherent granularity mismatch between modalities and leads to what we term semantic perceptual dissonance: highlevel abstract semantics provided by the LLM become inappropriately entangled with the lowlevel, fine-grained numerical dynamics of time series, making it difficult for semantic priors to effectively guide forecasting.To address this issue, we propose TimeSAF, a new framework based on hierarchical asynchronous fusion.Unlike synchronous approaches, TimeSAF explicitly decouples unimodal feature learning from cross-modal interaction.It introduces an independent cross-modal semantic fusion trunk, which uses learnable queries to aggregate global semantics from the temporal and prompt backbones in a bottom-up manner, and a stage-wise semantic refinement decoder that asynchronously injects these high-level signals back into the temporal backbone.This mechanism provides stable and efficient semantic guidance while avoiding interference with lowlevel temporal dynamics.Extensive experiments on standard long-term forecasting benchmarks show that TimeSAF significantly outperforms state-of-the-art baselines, and further exhibits strong generalization in both few-shot and zero-shot transfer settings.