The handwritten signature is the most used method of identity authentication. Due to their nature, signatures can be used as an agreement in many types of documentation with legal repercussions. The validation of the firmed signature is used to prevent frauds, fake documents, and identity checking. However, working with automated signature verification is a challenging task because it can appear in any part of documents with complex backgrounds, with logos, handwritten texts, and many different patterns. Besides, the application needs to consider a real-time response. In this paper, we propose an optimized architecture of a fully convolutional neural network based on the U-Net architecture for handwritten signature segmentation. Furthermore, we used data augmentation in order to increase the diversity of the available dataset and prevent the overfitting problem when training the proposed model. We conducted experiments with DSSigDataset, and we used four different data augmentation techniques to increase the dataset size. The experimental results show that our proposed approach speed-up the handwritten signature segmentation task, at the same time, achieving higher accuracy and lower variance than previous works.