This paper proposes a novel scheme for evaluating an emotional response to color images. The proposed scheme uses case-based reasoning in which the prototypical color images for each emotion are stored as cases and are compared with the images to be evaluated. In the comparison, the similarities in terms of image descriptors play an important role, and their combination is crucial for the construction of a proper similarity measure. In the training phase of the proposed scheme, the weights that represent the unequal importance of each descriptor is determined in order to obtain a similarity measure that can be used to evaluate and classify a color image with respect to a pair of emotions. Prior to classification, the representative color images are chosen for each emotion by human subjects and are stored as cases. The stored images are compared with an image to be classified using the constructed similarity measure to determine which emotion is appropriate between a pair of emotions. In this study, we used color and texture descriptors recommended by MPEG-7, represented as high-dimensional vectors. In the training, we proposed a method based on the rough approximation and the fuzzy inter- and intra-similarities to determine the weights that represent the unequal importance of the complex MPEG-7 descriptors. Experimental results show a promising performance for the proposed scheme, and better performance could be achieved by including more prototypical images as cases.