This paper proposes a novel automatic reference color selection (ARCS) scheme for the adaptive mathematical morphology (MM) method, and is specifically designed for color image segmentation applications. Because of the main advantages of being intuitive and simple, in the past decade, it has contributed to the growing popularity of binary and gray-scale MM processing. However, the MM process typically neglects the details of reference color determination. Applying other ordering methods, which select only black as the reference color for sorting pixels, result in the problem in which the scope of the distance measurement is not optimal. The proposed ARCS scheme is used for determining the ideal reference color for MM and for color image segmentation application. In addition, we use both 1D histogram-based modeling scheme binning from 3D color spaces, such as red-green-blue and hue-saturation-intensity, and 2D color models, such as (H, S), (Cb, Cr), and (I, By). According to the results of the quartile analysis, the threshold determination reacts with less sensitivity to the context variations of the images tested. The experiments focused on color-based image segmentation using the proposed ARCS scheme for color MM processing through a bottom-up scenario. To evaluate the system, four quantitative indices were utilized for an ARCS comparison using advanced segmentation methods in the experiments. The cross validation with different system parameters and a comparison of the morphological gradient operation with different color models are also presented.