On the basis of the maximum volume criterion in the hyperspectral imagery, the fast Gram determinant-based endmember extraction algorithm (FGDA) has the characteristics of simplicity, low computational complexity and no dimensionality reduction needed. However, the inversion matrix needs to be calculated in the procedure of FGDA, which still leads to the expensively computational cost as the number of extracted endmembers grows. The block matrix inversion lemma is used to solve the problem due to the symmetry property of the endmember Gram matrix. The improved approach can raise the efficiency through taking advantages of the recursive relationship of the block inversion matrix. A sub-scene of airborne visible/infrared imaging spectrometer (AVIRIS) hyperspectral data collected over Cuprite mining site, Nevada, is used to validate the method and the results of different initial conditions for FGDA are also analyzed. Experimental results demonstrate FGDA is  affected by its initial condition, and the proposed approach can improve the computational efficiency of FGDA as the number of endmembers and pixels in hyperspectral imagery increases.