This work is concerned with the binarization of ancient manuscripts that have been imaged with a MultiSpectral Imaging (MSI) system. We introduce a new dataset for this purpose that is composed of 130 multispectral images taken from two medieval manuscripts. We propose to apply an end-to-end Convolutional Neural Network (CNN) for the segmentation of the historical writings. The performance of the CNN based method is superior compared to two state-of-the-art methods that are especially designed for multispectral document images. The CNN based method is also evaluated on a previous and smaller database, where its performance is slightly worse than the two state-of-the-art techniques.