Abstract: In this study, the bag-of-visual-word based medical image classification technique was investigated. A new approach for medical image classification was proposed by introducing multi steps image classification using three different visual vocabularies. First, image is classified into general category by constructing that level of vocabulary. In second step image is classified into middle level by building another type of vocabulary and in last step specific vocabulary is calculated to perform exact classification. The proposed algorithm was evaluated on IRMA 2005 database consisting of 9,000 medical x-ray images of 57 classes. The accuracy rates obtained from three vocabularies are 95, 92 and 90%.