By the finite state machine learning algorithm, the state machine model of transport layer security (TLS) implementations could be extracted to analyze its security. At present, in the state machine learning, it is necessary to solve the problem that the state machine learning time increases exponentially as the number of target system states increases. An improved state machine learning algorithm is proposed, which used the TLS implementations’ specific sockets to reduce the required test sequence. It combined the checkpoint algorithm to construct the trie (i.e., prefix tree) of the test sequence, simplifying the test procedure of testing the test sequence. The test results showed that the proposed method can greatly reduce the number of equivalence queries generated by the state machine learning process, therefore accelerate the state machine learning process. At the same time, an abnormal state is analyzed through the learned state machine model, and a logic flaw of OpenSSl is found, which proved that the learned model is effective.