When the shipborne or airborne optoelectronic sensors sways, drop frames or the target makes complex tactical maneuvers, the tracking target will suddenly change its original trajectory between adjacent frames. How to effectively track the abrupt maneuvering target is a difficult problem. Firstly, a two frame difference method based on SURF feature descriptor is used for background subtraction, and then the predicted position of the target is given by the Kalman filter. In the searching area around this position, the tracking method of Mean shift is used to find the best matching of the target. Meanwhile, the frame by frame prediction error covariance of the Kalman filter is obtained for determining whether the target is maneuvering. After detecting the target maneuver, all suspicious targets in the field of view are quickly detected by the efficient sub-window search method based on saliency density. Finally, the original tracking target is selected by feature matching by using the SURF algorithm and the target position is returned to realize automatic and reliable tracking of abrupt maneuvering targets. The experimental results show that the new system can guarantee fast and accurate tracking effect for both conventional moving targets and abrupt maneuvering targets.