Accurate polyp segmentation of colonoscopy images is crucial for diagnosing and treating colorectal cancer. A new type of method for this challenging task has been introduced with the emergence of large vision models like the Segment Anything Model (SAM). However, the application of SAM to medical image segmentation falls short of clinical requirements. In efforts to tailor SAM for medical imaging tasks, researchers often work on improving adapters and prompts separately, over-looking the potential for synergy between them. Furthermore, existing methods still do not overcome the influence of prompt type, quantity, and positioning on segmentation results. SAM’s dependency on professionally annotated prompts further limits its utility. To tackle these issues, we propose a fine-tuning strategy named Automatic Prompt Tuning (APT), which aims to generate prompts automatically. This approach begins with adapting SAM for polyp segmentation by incorporating lightweight adapters into the image encoder. Next, we introduce an automatic prompt generator to process features and produce relevant prompts. Finally, we ensure the mask decoder consistently delivers accurate segmentation by leveraging these automatically generated prompts. Extensive experiments on three major polyp segmentation datasets demonstrate that APT outperforms other state-of-the-art approaches, proving its effectiveness.