This paper proposes an adaptive color feature extraction scheme by considering the color distribution of an image. Based on the binary quaternion-moment-preserving (BQMP) thresholding technique, the proposed extraction methods, fixed cardinality (FC) and variable cardinality (VC), are able to extract color features by preserving the color distribution of an image up to the third moment and to substantially reduce the distortion incurred in the extraction process. In addition to utilizing the earth mover's distance (EMD) as the distance measure of our color features, we also devise an efficient and effective distance measure, comparing histograms by clustering (CHIC). Moreover, the efficient implementation of our extraction methods is explored. With slight modification of the BQMP algorithm, our extraction methods are equipped with the capability of exploiting the concurrent property of hardware implementation. The experimental results show that our hardware implementation can achieve approximately a second order of magnitude improvement over the software implementation. It is noted that minimizing the distortion incurred in the extraction process can enhance the accuracy of the subsequent various image applications, and we evaluate the meaningfulness of the new extraction methods by the application to content-based image retrieval (CBIR). Our experimental results show that the proposed extraction methods can enhance the average retrieval precision rate by a factor of 25% over that of a traditional color feature extraction method.