Abstract: In order to enhance the robustness of tracking fast moving IR small target when we extract grey scale and local standard deviation of pixel as feature vector and treat tracking as two classification problems in target and local background feature vector pattern recognition, a new target tracking algorithm based on local background feature vectors Gaussian Mixture Model (GMM) clustering can be proposed. The method combines k-mean clustering with Expectation Maximization (EM) clustering algorithm, so the probability density parameters of the model are precisely determined and the GMM modeling speed is improved which are both important for real-time tracking. At the same time we can reform the model by using target feature vectors to improve the classification capability between the feature vectors of the target and background and then the classification model of the target and background is constructed. In the process, the Weighted Information Entropy (WIE) is applied as the discrimination criterion of the local background complexity and it can be used to guide the updating of the model adaptively. The validity of this algorithm is verified by the actual experiment.