Content-based retrieval (CBIR) methods in medical databases have been designed to support specific tasks, such as retrieval of digital mammograms or 3D MRI images. These methods cannot be transferred to other medical applications since different imaging modalities require different types of processing. To enable content-based queries in diverse collections of medical images, the retrieval system must be familiar with the current image class prior to the query processing. We describe a novel approach for the automatic categorization of medical images according to their modalities. We propose a semantically based set of visual features, their relevance and organization for capturing the semantics of different imaging modalities. The features are used in conjunction with a new categorization metric, enabling "intelligent" annotation, browsing/searching of medical databases. Our algorithm provides basic semantic knowledge about the image, and may serve as a front-end to the domain specific medical image analysis methods. To demonstrate the effectiveness of our approach, we have designed and implemented an Internet portal for browsing/querying online medical databases, and applied it to a large number of images. Our results demonstrate that accurate categorization can be achieved by exploiting the important visual properties of each modality.