In order to overcome the initial seed point selection and less robust of the order growth in the general region growing algorithm, the color image region growing algorithm is proposed with a robust order growth in this study. First, the local color histograms of all the pixels and Neighbor Similarity Factor (NSF) are calculated. Secondly, the seed selection rules, seed growth criteria and growth termination criterion are established by NSF value, segmentation is early made for image. Finally, unclassified points are reclassified to get the final segmentation result. By comparison with the JSEG algorithm testing, computation time and accuracy of the segmentation algorithm has obvious advantages.