The widespread adoption of online Large Language Models (LLMs) raises considerable privacy concerns, as prompts may inadvertently contain sensitive information exposed to LLM service providers. Limited by high computational costs, reduced response utility, and excessive system modifications, previous works based on local deployment, embedding perturbation, and homomorphic encryption are not feasible for online prompt-based LLM services. To address these issues, we introduce ProSan (Prompt Privacy Sanitizer), an end-to-end method for prompt privacy protection that generates prompts with task-irrelevant privacy removed, while preserving both utility and readability. It can also be seamlessly integrated into the online LLM service pipeline. To achieve high utility and contextual privacy, ProSan flexibly adjusts its protection targets and strength based on the importance of the words and the privacy leakage risk of the prompts. Additionally, ProSan is capable of adapting to diverse computational resource conditions, ensuring privacy protection for low-resource users. Our experiments demonstrate that ProSan effectively removes sensitive information across various tasks, including question answering, text summarization, and code generation, with minimal reduction in task performance.