Abstract: Based on the assumption that background appears with large appearance frequency, a new background reconstruction algorithm based on basic sequential clustering is proposed in this research. First, pixel intensity in period of time are classified based on mend basic sequential clustering. Second, merging procedure and reassignment procedure are run to classified classes. Finally, pixel intensity classes, whose appearance frequencies are higher than a threshold, are selected as the background pixel intensity value. So the improved algorithm can rebuilt the background images of various scenes. Compared with the background reconstruction method based on basic sequential clustering, the simulation results show that those near classes are avoided at all and the effect of input order of data has been reduced greatly in our method. And the background model can represent the scene well.