Next Point-of-Interest (POI) recommendation aims to predict user's subsequent destinations based on historical check-in sequences, thereby enhancing travel experiences. While traditional methods primarily rely on unique identifiers (IDs) to represent POIs, they face data scarcity challenges. Recent multi-modal approaches offer alternatives but struggle with two key issues: inadequate handling of heterogeneity between ID and multi-modal features, and difficulties in unified framework integration, limiting their potential benefits. To address these limitations, we propose IM-POI, a novel framework that leverages the complementary strengths of both ID embeddings and multi-modal representations for next POI recommendation. In our framework, a global POI weighted transition graph inspired by TF-IDF captures sequential dependencies and enhances memorization capabilities, while a geographical graph incorporates spatial information into multi-modal features to be consistent with real-world visitation patterns. To address representation integration, we introduce an IM-Aligner module to prevent representation collapse during distribution matching. Extensive experiments on three real-world datasets demonstrate that IM-POI significantly outperforms state-of-the-art baselines.