Traditional methods of time delay estimation are confronted with the basic contradiction between accuracy and resolution, and it is difficult to obtain high accuracy and unambiguous estimation at once. To deal with this problem, a two-stage high accuracy and unambiguous time delay estimation method based on interferometric phase is proposed. The Kalman filter is used to enhance the performance of resistance to noise for the interferometric phase after obtaining the cross spectrum of the signal, and then get the unambiguous but low accuracy estimate at large frequency aperture, which is regarded as the first step. The second step is to solve the ambiguity of the phase delay by all the frequencies in the given frequency span using the estimate obtained in the first step, which greatly increases spectrum utilization. Moreover, since the same time delay is measured by different frequencies, statistical processing can be performed on the results, which further improves the estimation accuracy. Simulations verify the validity of the proposed method. For signals with bandwidths of 10 MHz, the accuracy is better than 1ps when the SNR is higher than 18 dB.