Large language models (LLMs) are increasingly integrated into users' daily lives, leading to a growing demand for personalized outputs.Previous work focuses on leveraging a user's own history, overlooking inter-user differences that are crucial for effective personalization.While recent work has attempted to model such differences, the reliance on language-based prompts often hampers the effective extraction of meaningful distinctions.To address these issues, we propose Difference-aware Embeddingbased Personalization (DEP), a framework that models inter-user differences in the latent space instead of relying on language prompts.DEP constructs soft prompts by contrasting a user's embedding with those of peers who engaged with similar content, highlighting relative behavioral signals.A sparse autoencoder then filters and compresses both user-specific and difference-aware embeddings, preserving only task-relevant features before injecting them into a frozen LLM.Experiments on personalized review generation show that DEP consistently outperforms baseline methods across multiple metrics.