受限玻尔兹曼机及其变体研究综述

doi:10.12305/j.issn.1001-506X.2024.07.16

Abstract

Abstract:

As a typical probabilistic graphical model for learning data distribution and extracting intrinsic features, the restricted Boltzmann machine (RBM) is an important fundamental model in the field of deep learning. In recent years, numerous emerging models, i.e., RBM variants, have been obtained by improving the model structure and energy function of RBM, which can further enhance the feature extraction performance of the model. The study of RBM and its variants can significantly contribute to the development of the deep learning field and realize the rapid extraction of massive information in the era of big data. Based on this, the relevant research on RBM and its variants are systematically reviewed in recent years, and the improvement of training algorithm, model structure, deep model fusion research and the latest application are creatively reviewed. In particular, the focus is on sorting out the develop history of training algorithms and variants for RBM. Finally, the existing difficulties and challenges in the field of RBM and its variants are discussed, and the main research work is summarized and prospected.

Key words: restricted Boltzmann machine (RBM), deep learning, restricted Boltzmann machine variants, probabilistic undirected graph, feature extraction

CLC Number:

TP183

Qianglong WANG, Xiaoguang GAO, Bicong WU, Zijian HU, Kaifang WAN. Review of research on restricted Boltzmann machine and its variants[J]. Systems Engineering and Electronics, 2024, 46(7): 2323-2345.

Figures/Tables 25

Fig.1

Fig.2

Fig.3

Fig.4

Table 1

Fig.5

Table 2

Fig.6

Fig.7

Fig.8

Fig.9

Fig.10

Fig.11

Fig.12

Fig.13

Fig.14

Fig.15

Fig.16

Fig.17

Fig.18

Fig.19

Table 3

Table 4

Fig.20

Table 5

References 124

1	GOODFELLOW I , BENGIO Y , COURVILLE A . Deep learning[M]. Massachusetts: Massachusetts Institute of Technology, 2017.
2	HINTON G E , SALAKHUTDINOV R R . Reducing the dimensionality of data with neural networks[J]. Science, 2006, 313 (5786): 504- 507. doi: 10.1126/science.1127647
3	HINTON G E , OSINDERO S , TEH Y W . A fast learning algorithm for deep belief nets[J]. Neural Computation, 2006, 18 (7): 1527- 1554. doi: 10.1162/neco.2006.18.7.1527
4	REICHERT D P, SERIES P, STORKEY A J. Hallucinations in charles bonnet syndrome induced by homeostasis: a deep Boltzmann machine model[C]//Proc. of the Neural Information Processing Systems Conference, 2010.
5	CHO K H, RAIKO T, ILIN A. Gaussian Bernoulli deep Boltzmann machine[C]//Proc. of the International Joint Conference on Neural Networks, 2013.
6	SCHOLKOPF B, PLATT J, HOFMANN T. Greedy layer-wise training of deep networks[C]//Proc. of the Neural Information Processing Systems Conference, 2007, 19: 153-160.
7	胡铭菲, 左信, 刘建伟. 深度生成模型综述[J]. 自动化学报, 2022, 48 (1): 40- 74.
	HU M F , ZUO X , LIU J W . Survey on deep generative model[J]. Acta Automatica Sinica, 2022, 48 (1): 40- 74.
8	FISCHER A . Training restricted Boltzmann machines[J]. Kunstliche Intelligenz, 2015, 29 (4): 441- 444. doi: 10.1007/s13218-015-0371-2
9	HINTON G E . Training products of experts by minimizing contrastive divergence[J]. Neural Computation, 2002, 14 (8): 1771- 1800. doi: 10.1162/089976602760128018
10	TIELEMAN T. Training restricted Boltzmann machines using approximations to the likelihood gradient[C]//Proc. of the International Conference on Machine Learning, 2008.
11	TIELEMAN T, HINTON G E. Using fast weights to improve persistent contrastive divergence[C]//Proc. of the 26th International Conference on Machine Learning, 2009.
12	CHO K H, RAIKO T, ILIN A. Parallel tempering is efficient for learning restricted Boltzmann machines[C]//Proc. of the International Joint Conference on Neural Networks, 2012.
13	DESJARDINS G, COURVILLE A C, BENGIO Y, et al. Tempered Markov chain Monte Carlo for training of restricted Boltzmann machines[C]//Proc. of the International Conference on Artificial Intelligence and Statistics, 2010.
14	DESJARDINS G, COURVILLE A, BENGIO Y, et al. Parallel tempering for training of restricted Boltzmann machines[C]//Proc. of the International Conference on Artificial Intelligence and Statistics, 2010.
15	李飞, 高晓光, 万开方. 基于动态Gibbs采样的RBM训练算法研究[J]. 自动化学报, 2016, 42 (6): 931- 942.
	LI F , GAO X G , WAN K F . Research on RBM training algorithm with dynamic Gibbs sampling[J]. Acta Automatica Sinica, 2016, 42 (6): 931- 942.
16	LI X Y , GAO X G , WANG C F . A novel restricted Boltzmann machine training algorithm with dynamic tempering chains[J]. IEEE Access, 2021, 9, 521939- 21950.
17	WANG Q L , GAO X G , WAN K F , et al. A novel restricted Boltzmann machine training algorithm with fast Gibbs sampling policy[J]. Mathematical Problems in Engineering, 2020, 4206457.
18	MA X S , WANG X J . Average contrastive divergence for training restricted Boltzmann machines[J]. Entropy, 2016, 18 (1): 18- 32.
19	KRAUSE O , FISCHER A , IGEL C . Population-contrastive-divergence: does consistency help with RBM training?[J]. Pattern Recognition Letters, 2018, 102, 1- 7. doi: 10.1016/j.patrec.2017.11.022
20	DU Y L, LI S, TENENBAUM J, et al. Improved contrastive divergence training of energy-based models[C]//Proc. of the International Conference on Machine Learning, 2021.
21	NING L , PITTMAN R , SHEN X P . LCD: a fast contrastive divergence based algorithm for restricted Boltzmann machine[J]. Neural Networks, 2018, 108, 399- 410. doi: 10.1016/j.neunet.2018.08.018
22	ROMERO E , MAZZANTI F , DELGADO J , et al. Weighted contrastive divergence[J]. Neural Networks, 2019, 114, 147- 156. doi: 10.1016/j.neunet.2018.09.013
23	RUIZ F J R, TITSIAS M K. A contrastive divergence for combining variational inference and MCMC[C]//Proc. of the International Conference on Machine Learning, 2019.
24	PEDRONI B U, SHEIK S, CAUWENBERGHS G. Pipelined parallel contrastive divergence for continuous generative model learning[C]//Proc. of the IEEE International Symposium on Circuits and Systems, 2017: 212-215.
25	SANJAYA P , KANG D K . Optimizing restricted Boltzmann machine learning by injecting Gaussian noise to likelihood gradient approximation[J]. Applied Intelligence, 2019, 49 (7): 2723- 2734. doi: 10.1007/s10489-018-01400-5
26	CHO K , RAIKO T , ILIN A . Enhanced gradient for training restricted Boltzmann machines[J]. Neural Computation, 2013, 25 (3): 805- 831. doi: 10.1162/NECO_a_00397
27	李飞, 高晓光, 万开方. 基于权值动量的RBM加速学习算法研究[J]. 自动化学报, 2017, 43 (7): 1142- 1159.
	LI F , GAO X G , WAN K F . Research on RBM accelerating learning algorithm with weight momentum[J]. Acta Automatica Sinica, 2017, 43 (7): 1142- 1159.
28	李飞, 高晓光, 万开方. 基于改进并行回火算法的RBM网络训练研究[J]. 自动化学报, 2017, 43 (5): 753- 764.
	LI F , GAO X G , WAN K F . Research on RBM networks training based on improved parallel tempering algorithm[J]. Acta Automatica Sinica, 2017, 43 (5): 753- 764.
29	沈卉卉, 李宏伟. 基于动量方法的受限玻尔兹曼机的一种有效算法[J]. 电子学报, 2019, 47 (1): 176- 182. doi: 10.3969/j.issn.0372-2112.2019.01.023
	SHEN H H , LI H W . An efficient algorithm on restricted Boltzmann machines based on momentum method[J]. Acta Electronica Sinica, 2019, 47 (1): 176- 182. doi: 10.3969/j.issn.0372-2112.2019.01.023
30	SHEN H H , LI H W . A gradient approximation algorithm based weight momentum for restricted Boltzmann machine[J]. Neurocomputing, 2019, 361, 40- 49. doi: 10.1016/j.neucom.2019.07.074
31	WANG Q L , GAO X G , LI X Y , et al. A precise method for RBMs training using phased curricula[J]. Multimedia Tools and Applications, 2022, 82 (6): 8013- 8047.
32	YIN J, MAO Q Y, LIU D Y H, et al. Method to improve the performance of restricted Boltzmann machines[C]//Proc. of the International Symposium on Neural Networks, 2018: 115-123.
33	YIN J , LV J C , SANG Y S , et al. Classification model of restricted Boltzmann machine based on reconstruction error[J]. Neural Computing & Applications, 2018, 29 (11): 1171- 1186.
34	GOLUBEVA A , MELKO R G . Pruning a restricted Boltzmann machine for quantum state reconstruction[J]. Physical Review B, 2022, 105 (12): 125124. doi: 10.1103/PhysRevB.105.125124
35	DIXIT V , SELVARAJAN R , ALAM M A , et al. Training restricted Boltzmann machines with a D-wave quantum annealer[J]. Frontiers in Physics, 2021, 9, 589626. doi: 10.3389/fphy.2021.589626
36	RODER M, DE ROSA G H, PAPA J P, et al. Fine-tuning temperatures in restricted Boltzmann machines using meta-heuristic optimization[C]//Proc. of the IEEE World Congress on Computational Intelligence, 2020.
37	TAKAMUNE N, KAMEOKA H. Maximum reconstruction probability training of restricted Boltzmann machines with auxiliary function approach[C]//Proc. of the IEEE International Workshop on Machine Learning for Signal Processing, 2014.
38	LUO L K, WANG Y D, PENG H, et al. Training restricted Boltzmann machine with dynamic learning rate[C]//Proc. of the International Conference on Computer Science & Education, 2016: 103-107.
39	WELLING M. A new learning algorithm for mean field Boltzmann machines[C]//Proc. of the International Conference on Artifical Neural Networks, 2002.
40	张健. 基于RBM的深度神经网络算法研究[D]. 北京: 中国矿业大学, 2020.
	ZHANG J. Research on deep neural networks based on RBM[D]. Beijing: China Mining University, 2020.
41	LAROCHELLE H , MANDEL M , PASCANU R , et al. Learning algorithms for the classification restricted Boltzmann machine[J]. Journal of Machine Learning Research, 2012, 13 (1): 643- 669.
42	SRIVASTAVA N , HINTON G , KRIZHEVSKY A , et al. Dropout: a simple way to prevent neural networks from overfitting[J]. Journal of Machine Learning Research, 2014, 15 (1): 1929- 1958.
43	LI W, ZEILER M, ZHANG S, et al. Regularization of neural networks using dropconnect[C]//Proc. of the International Conference on Machine Learning, 2013: 2095-2103.
44	TAYLOR G W, HINTON G E, ROWEIS S T. Modeling human motion using binary latent variables[C]//Proc. of the Advances in Neural Information Processing Systems (NIPS), 2007.
45	SUTSKEVER I, HINTON G E, TAYLOR G W. The recurrent temporal restricted Boltzmann machine[C]//Proc. of the Neural Information Processing Systems Conference, 2008.
46	COTE M A , LAROCHELLE H . An infinite restricted Boltzmann machine[J]. Neural Computation, 2016, 28 (7): 1265- 1288. doi: 10.1162/NECO_a_00848
47	WANG Q L , GAO X G , WAN K F , et al. Generative and discriminative infinite restricted Boltzmann machine training[J]. International Journal of Intelligent Systems, 2022, 37 (10): 7857- 7887. doi: 10.1002/int.22908
48	YUN J, XIAO J Z, XI L, et al. A removing redundancy restricted Boltzmann machine[C]//Proc. of the 10th International Conference on Advanced Computational Intelligence, 2018.
49	LU X Q , MENG L Z , CHEN C , et al. Fuzzy removing redundancy restricted Boltzmann machine: improving learning speed and classification accuracy[J]. IEEE Trans.on Fuzzy Systems, 2020, 28 (10): 2495- 2509.
50	NGUYEN T D , TRAN T , PHUNG D , et al. Graph-induced restricted Boltzmann machines for document modeling[J]. Information Sciences, 2016, 328, 60- 75. doi: 10.1016/j.ins.2015.08.023
51	ZHANG N , DING S F , LIAO H M , et al. Multimodal correlation deep belief networks for multi-view classification[J]. Appllied Intelligence, 2019, 49, 1925- 1936. doi: 10.1007/s10489-018-1379-8
52	ZHANG N , DING S F , SUN T F , et al. Multi-view RBM with posterior consistency and domain adaptation[J]. Information Sciences: an International Journal, 2019, 516 (5786): 142- 157.
53	ZHANG N , SUN S L . Multiview graph restricted Boltzmann machines[J]. IEEE Trans.on Cybernetics, 2022, 52 (11): 12414- 12428. doi: 10.1109/TCYB.2021.3084464
54	RODER M , DE-ROSA G H , ALBUQUERQUE V H C , et al. Energy-based dropout in restricted Boltzmann machines: why not go random[J]. IEEE Trans.on Emerging Topics in Computational Intelligence, 2022, 6 (2): 276- 286. doi: 10.1109/TETCI.2020.3043764
55	TRAN S N , GARCEZ A D , WEYDE T , et al. Sequence classification restricted Boltzmann machines with gated units[J]. IEEE Trans.on Neural Networks and Learning Systems, 2020, 31 (11): 4806- 4815. doi: 10.1109/TNNLS.2019.2958103
56	NAKASHIKA T, TAKAKI S, YAMAGISHI J. Complex-valued restricted Boltzmann machine for direct learning of frequency spectra[C]//Proc. of the 18th Annual Conference of the International Speech Communication Association, 2017.
57	CHO K H, ILIN A, RAIKO T. Improved learning of Gaussian-Bernoulli restricted Boltzmann machines[C]//Proc. of the International Conference on Artificial Neural Networks, 2011: 10-17.
58	RANZATO M, KRIZHEVSKY A, HINTON G E. Factored 3-way restricted Boltzmann machines for modeling natural images[C]//Proc. of the International Conference on Artificial Intelligence and Statistics, 2010.
59	RANZATO M, HINTON G E. Modeling pixel means and covariances using factorized third-order Boltzmann machines[C]//Proc. of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2010.
60	COURVILLE A, BERGSTRA J, BENGIO Y. A spike and slab restricted Boltzmann machine[C]//Proc. of the International Conference on Artificial Intelligence and Statistics, 2011.
61	COURVILLE A, BERGSTRA J, BENGIO Y. Unsupervised models of images by spike-and-slab RBMs[C]//Proc. of the International Conference on International Conference on Machine Learning, 2011.
62	ZHANG J , WANG H J , CHU J L , et al. Improved Gaussian-Bernoulli restricted Boltzmann machine for learning discriminative representations[J]. Knowledge-Based System, 2019, 185, 104911. doi: 10.1016/j.knosys.2019.104911
63	CHEN C L P , ZHANG C Y , CHEN L , et al. Fuzzy restricted Boltzmann machine for the enhancement of deep learning[J]. IEEE Trans.on Fuzzy Systems, 2015, 23 (6): 2163- 2173. doi: 10.1109/TFUZZ.2015.2406889
64	HOSSEINI-POZVEH M S , SAFAYANI M , MIRZAEI A . Interval type-2 fuzzy restricted Boltzmann machine[J]. IEEE Trans.on Fuzzy Systems, 2021, 29 (5): 1133- 1144. doi: 10.1109/TFUZZ.2020.2969120
65	HARRISON R W . Continuous restricted Boltzmann machines[J]. Wireless Networks, 2022, 28 (3): 1263- 1267. doi: 10.1007/s11276-018-01903-6
66	NAIR V, HINTON G E. Rectified linear units improve restricted Boltzmann machines[C]//Proc. of the International Conference on International Conference on Machine Learning, 2010.
67	RAVANBAKHSH S, POCZOS B, SCHNEIDER J, et al. Stochastic neural networks with monotonic activation functions[EB/OL]. [2023-07-20]. https://arxiv.org/pdf/1601.00034.pdf.
68	LI Y F, ZHU X D. Exponential family restricted Boltzmann machines and annealed importance sampling[C]//Proc. of the International Joint Conference on Neural Networks, 2018: 39-48.
69	ZHANG J , DING S F , ZHANG N . An overview on probability undirected graphs and their applications in image processing[J]. Neurocomputing, 2018, 321, 156- 168. doi: 10.1016/j.neucom.2018.07.078
70	LU W H , LEUNG C S , SUM J . Analysis on noisy Boltzmann machines and noisy restricted Boltzmann machines[J]. IEEE Access, 2021, 9, 112955- 112965. doi: 10.1109/ACCESS.2021.3102275
71	FISHER C K, SMITH A M, WALSH J R. Boltzmann encoded adversarial machines[EB/OL]. [2023-07-20]. https://arxiv.org/abs/1804.08682.
72	MONTAVON G, MUELLER K, CUTURI M. Wasserstein training of restricted Boltzmann machines[C]//Proc. of the Neural Information Processing Systems Conference, 2016.
73	DEMERTZIS K , ILIADIS L , PIMENIDIS E , et al. Variational restricted Boltzmann machines to automated anomaly detection[J]. Neural Computing & Applications, 2022, 34 (18): 15207- 15220.
74	SALAKHUTDINOV R, HINTON G. A better way to pretrain deep Boltzmann machines[C]//Proc. of the Neural Information Processing Systems Conference, 2012.
75	SALAKHUTDINOV R, HINTON G E. Deep Boltzmann machines[C]//Proc. of the International Conference on Artificial Intelligence and Statistics, 2009.
76	SALAKHUTDINOV R , HINTON G . An efficient learning procedure for deep Boltzmann machines[J]. Neural Computation, 2012, 24 (8): 1967- 2006. doi: 10.1162/NECO_a_00311
77	GOODFELLOW I J, MIRZA M, COURVILLE A, et al. Multi-prediction deep Boltzmann machines[C]//Proc. of the Neural Information Processing Systems Conference, 2013.
78	SHUANG F, CHEN C L P. Fuzzy restricted Boltzmann machine and deep belief network: a comparison on image reconstruction[C]//Proc. of the IEEE International Conference on Systems, Man, and Cybernetics, 2017: 1828-1833.
79	WANG G M , QIAO J F , BI J , et al. An adaptive deep belief network with sparse restricted Boltzmann machines[J]. IEEE Trans.on Neural Networks and Learning Systems, 2020, 31 (10): 4217- 4228. doi: 10.1109/TNNLS.2019.2952864
80	VERA M , VEGA L R , PIANTANIDA P . Information flow in deep restricted Boltzmann machines: an analysis of mutual information between inputs and outputs[J]. Neurocomputing, 2022, 507, 235- 246. doi: 10.1016/j.neucom.2022.08.014
81	AGLIARI E , SEBASTIANI G . Learning and retrieval operational modes for three-layer restricted Boltzmann machines[J]. Journal of Statistical Physics, 2021, 185 (2): 10. doi: 10.1007/s10955-021-02841-y
82	LEONELLI F E , AGLIARI E , ALBANESE L , et al. On the effective initialisation for restricted Boltzmann machines via duality with hopfield model[J]. Neural Networks, 2021, 143, 314- 326. doi: 10.1016/j.neunet.2021.06.017
83	FENG S , CHEN C L P , ZHANG C Y . A fuzzy deep model based on fuzzy restricted Boltzmann machines for high-dimensional data classification[J]. IEEE Trans.on Fuzzy Systems, 2020, 28 (7): 1344- 1355.
84	KANG T Y, CHEN P, QUACKENBUSH J, et al. A novel deep learning model by stacking conditional restricted Boltzmann machine and deep neural network[C]//Proc. of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2017: 1828-1833.
85	LI C P , WANG J L , YE X Z . Using a recurrent neural network and restricted Boltzmann machines for malicious traffic detection[J]. Neuroquantology, 2018, 16 (5): 823- 831.
86	LU F , WU J D , HUANG J Q , et al. Restricted-Boltzmann-based extreme learning machine for gas path fault diagnosis of turbofan engine[J]. IEEE Trans.on Industrial Informatics, 2020, 16 (2): 959- 968. doi: 10.1109/TII.2019.2921032
87	WU X J , XU M D , LI C D , et al. Research on image reconstruction algorithms based on autoencoder neural network of restricted Boltzmann machine (RBM)[J]. Flow Measurement and Instrumentation, 2021, 80, 102009. doi: 10.1016/j.flowmeasinst.2021.102009
88	SOBCZAK S , KAPELA R . Hybrid restricted Boltzmann machine-convolutional neural network model for image recognition[J]. IEEE Access, 2022, 10, 24985- 24994. doi: 10.1109/ACCESS.2022.3155873
89	HAN Z Z , LIU Z B , HAN J W , et al. Mesh convolutional restricted Boltzmann machines for unsupervised learning of features with structure preservation on 3-D meshes[J]. IEEE Trans.on Neural Networks and Learning Systems, 2017, 28 (10): 2268- 2281. doi: 10.1109/TNNLS.2016.2582532
90	KARTHIK E , SETHUKARASI T . A centered convolutional restricted Boltzmann machine optimized by hybrid atom search arithmetic optimization algorithm for sentimental analysis[J]. Neural Processing Letters, 2022, 54 (5): 4123- 4151. doi: 10.1007/s11063-022-10797-7
91	KANNO Y, YASUDA M. Multi-layered discriminative restricted Boltzmann machine with untrained probabilistic layer[C]//Proc. of the International Conference on Pattern Recognition, 2021: 7655-7660.
92	VAN-TULDER G , DE-BRUIJNE M . Combining generative and discriminative representation learning for lung CT analysis with convolutional restricted Boltzmann machines[J]. IEEE Trans.on Medical Imaging, 2016, 35 (5): 1262- 1272. doi: 10.1109/TMI.2016.2526687
93	CHEN C L P , FENG S . Generative and discriminative fuzzy restricted Boltzmann machine learning for text and image classification[J]. IEEE Trans.on Cybernetics, 2020, 50 (5): 2237- 2248. doi: 10.1109/TCYB.2018.2869902
94	LUO L K , ZHANG S F , WANG Y D , et al. An alternate method between generative objective and discriminative objective in training classification restricted Boltzmann machine[J]. Knowledge Based Systems, 2018, 144, 144- 152. doi: 10.1016/j.knosys.2017.12.032
95	XU A Q , TIAN M W , FIROUZI B , et al. A new deep learning restricted Boltzmann machine for energy consumption forecasting[J]. Sustainability-Basel, 2022, 14 (16): 10081. doi: 10.3390/su141610081
96	JUSTIN S , SALEH W , AL GHAMDI T , et al. Hyper parameter optimization based deep belief network for clean buses using solar energy model[J]. Intelligent Automation and Soft Computing, 2023, 37 (1): 1091- 1109. doi: 10.32604/iasc.2023.032589
97	WANG Q , YANG X L , PU D , et al. Sustainable investment forecasting of power grids based on the deep restricted Boltzmann machine optimized by the lion algorithm[J]. Computer Modeling in Engineering and Sciences, 2022, 130 (1): 269- 286. doi: 10.32604/cmes.2022.016437
98	SHIN U , ISHⅡ M , OKAZAKI A , et al. Pattern training, inference, and regeneration demonstration using on-chip trainable neuromorphic chips for spiking restricted Boltzmann machine[J]. Advance Intelligent Systems, 2022, 4 (8): 2200034. doi: 10.1002/aisy.202200034
99	PATEL S , CANOZA P , SALAHUDDIN S . Logically synthesized and hardware accelerated restricted Boltzmann machines for combinatorial optimization and integer factorization[J]. Nature Electronics, 2022, 5 (2): 92- 101. doi: 10.1038/s41928-022-00714-0
100	YANG X F , WU Z , ZHANG Q M . Bluetooth indoor localization with Gaussian-Bernoulli restricted Boltzmann machine plus liquid state machine[J]. IEEE Trans.on Instrumentation and Measurement, 2022, 71, 1000208.
101	YUAN X H , WANG C , LI M Q , et al. Lapping quality prediction of ceramic fiber brush based on Gaussian-restricted Boltzmann machine[J]. Materials, 2022, 15 (21): 7805. doi: 10.3390/ma15217805
102	TRENTIN V , ARTUNEDO A , GODOY J , et al. Multi-modal interaction-aware motion prediction at unsignalized intersections[J]. IEEE Trans.on Intelligent Vehicles, 2023, 8 (5): 3349- 3365. doi: 10.1109/TIV.2023.3254657
103	ELANGOVAN D , SUBEDHA V . Adaptive particle grey wolf optimizer with deep learning-based sentiment analysis on online product reviews[J]. Engineering Technology & Applied Science Research, 2023, 13 (3): 10989- 10993.
104	SUN C X, WANG L H, WANG W, et al. Non-small cell lung cancer staging using deep restricted Boltzmann machine[C]//Proc. of the IEEE International Conference on Signal Processing, 2018: 1175-1178.
105	WAHID F , AZHAR S , ALI S , et al. Pneumonia detection in chest X-ray images using enhanced restricted Boltzmann machine[J]. Journal of Healthcare Engineering, 2022, 1678000.
106	QIAN Y Q , DING Y J , ZOU Q , et al. Identification of drug-side effect association via restricted Boltzmann machines with penalized term[J]. Brief Bioinform, 2022, 23 (6): 458. doi: 10.1093/bib/bbac458
107	NARAMALA V R , KUMAR B A , RAO V S , et al. Enhancing diabetic retinopathy detection through machine learning with restricted Boltzmann machines[J]. International Journal of Advanced Computer Science and Applications, 2023, 14 (9): 573- 585.
108	ALQAHTANI N , ALAM S , AQEEL I , et al. Deep belief networks (DBN) with IoT-based Alzheimer's disease detection and classification[J]. Applied Sciences-Basel, 2023, 13 (3): 7833.
109	LI J J , XU F M , GAO N , et al. Sparse non-convex regularization based explainable DBN in the analysis of brain abnormalities in schizophrenia[J]. Computers in Biology and Medicine, 2023, 155, 106664. doi: 10.1016/j.compbiomed.2023.106664
110	NAVEEN P . Occlusion-aware facial expression recognition: a deep learning approach[J]. Multimedia Tools and Applications, 2023, doi: 10.1007/s11042-023-17013-1
111	ZOU X C , WANG G J , YU G X . Protein function prediction using deep restricted Boltzmann machines[J]. Biomed Research International, 2017, 1729301.
112	SAHA S , BANDOPADHYAY S , GHOSH A . Identifying the degree of genetic interactions using restricted Boltzmann machine: a study on colorectal cancer[J]. IET Systems Biology, 2021, 15 (1): 26- 39. doi: 10.1049/syb2.12009
113	KUMAR R R S R , APPARAO G , ANURADHA S . Deep scalable and distributed restricted Boltzmann machine for recommendations[J]. International Journal of System Assurance Engineering and Management, 2022, doi: 10.1007/s13198-022-01684-4
114	JHA G K , GAUR M , RANJAN P , et al. A trustworthy model of recommender system using hyper-tuned restricted Boltzmann machine[J]. Multimedia Tools and Applications, 2022, 82 (6): 8261- 8285.
115	ROOHULLAH , WAHID F , ALI S , et al. Accident detection in autonomous vehicles using modified restricted Boltzmann machine[J]. Security and Communication Networks, 2022, 6424835.
116	LAKSHMI V M , RADHIKA R , KALPANA G . Radial restricted Boltzmann machines with functional neural network for classification of the fake and real news analysis[J]. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 2022, 30 (1): 31- 43.
117	LI J C , YU Z L , GU Z H , et al. Spatial-temporal discriminative restricted Boltzmann machine for event-related potential detection and analysis[J]. IEEE Trans.on Neural Systems and Rehabilitation Engineering, 2019, 27 (2): 139- 151. doi: 10.1109/TNSRE.2019.2892960
118	TANG Y , XIAO X , YANG X , et al. Research on a small sample feature transfer method for fault diagnosis of reciprocating compressors[J]. Journal of Loss Prevention in the Process Industries, 2023, 85, 105163. doi: 10.1016/j.jlp.2023.105163
119	LI B Q , WU J Y , LI L S , et al. Adaptive assessment of power system transient stability based on active transfer learning[J]. Automation of Electric Power Systems, 2023, 47 (4): 121- 132.
120	LI Y , PENG L H , LI J Q , et al. Fabric defect detection based on deep-belief network[J]. Journal of Textile Research, 2023, 44 (2): 143- 150.
121	WANG D Y , ZHOU L J , YANG Z X , et al. Defects detection of onboard cable termination in EMUs using partial discharge measurement and SDP-DL framework[J]. IEEE Trans.on Transportation Electrification, 2023, 9 (2): 3127- 3136. doi: 10.1109/TTE.2022.3225312
122	ASLAN N , DOGAN S , KOCA G . Automated classification of brain diseases using the restricted Boltzmann machine and the generative adversarial network[J]. Engineering Applications of Artificial Intelligence, 2023, 126 (Part A): 106794.
123	GUO H F . The evaluation of university management performance using the CS-RBM algorithm[J]. PEERJ Computer Science, 2023, 9, e1575.
124	ZHANG Y Y . Analysis of benefits and security of China's foreign investment with artificial intelligence under the belt and road[J]. Applied Artificial Intelligence, 2023, 37 (1): 2219562.

算法	采样链数	核心方法	算法说明	算法优势
CD^[9]	单条	应用Gibbs采样近似模型分布	Gibbs	简单快捷最常用的RBM训练算法
PCD^[10]	单条	使用持续采样链进行采样	Gibbs+持续链	减小CD算法的拟合偏差
FPCD^[11]	单条	使用加速权重	Gibbs+加速权重	加速持续链采样算法的收敛速度
PT^[12-14]	多条	加入温度机制, 多采样链并行操作有效交互	Gibbs+温度机制	能够处理多模分布的复杂数据
DGS^[15]	单条	动态改进采样链采样步数	Gibbs+动态机制	缩小CD算法有偏估计误差
DTC^[16]	多条	动态改进采样链条数	Gibbs+动态机制	缩小PT算法拟合与实际模型偏差
FGS^[17]	单条	启用加速参数和调整系数	Gibbs+加速参数	加速Gibbs链采样算法的采样速度
ACD^[18]	单条	使用均值优化参数更新梯度	Gibbs+优化梯度	提升CD算法的对数似然梯度
PopCD^[19]	单条	使用种群蒙特卡罗理论对样本数据进行重新加权	Gibbs+样本加权	效减少CD算法的有偏估计误差
ICD^[20]	单条	将数据增强手段应用到CD算法	Gibbs+增强梯度	生成高分辨率图像并具有鲁棒性
LCD^[21]	单条	给出神经元的条件概率上下限, 重复利用历史结果	Gibbs+去冗余机制	消除冗余计算并提升算法效率
WCD^[22]	单条	对CD算法负向梯度的不同状态进行不同的加权	Gibbs+优化梯度	近似梯度更接近于真实梯度
VCD^[23]	单条	结合马尔可夫蒙特卡罗和变分推理优势	Gibbs+变分推断	优化梯度更新计算方式
PPCD^[24]	单条	通过空间交换时间, 同时处理CD算法的两个阶段	Gibbs+时空机制	可以连续处理和学习时间流数据
NPCD^[25]	单条	使用额外的噪声扰动	Gibbs+抗干扰机制	加速采样链收敛, 提高算法鲁棒性

提出算法	实现机制及理论	算法改进	硬件架构
Wang等^[31]	课程学习融合	√	-
Yin等^[32-33]	目标函数改进	√	-
Golubeva等^[34]	神经网络裁剪	√	√
Dixit等^[35]	量子退火器训练	-	√
Roder等^[36]	粒子群算法优化	√	√
Takamune等^[37]	辅助函数优化	√	-
Luo等^[38]	动态学习率优化	√	-

RBM变体	改进方法	增节点	删节点	结构扩展	核心优势
DRBM^[41]	标签和数据输入联合训练	√	-	-	独立判别性模型
DrRBM^[42]	随机删除节点的正则化模型	-	√	-	有效预防过拟合
DcRBM^[43]	随机删除连接的正则化模型	-	√	-	有效预防过拟合
TRBM^[44]	多个RBM按照时序联合	-	-	√	能够处理时序数据
RTRBM^[45]	循环结构, 使用之前模型信息	-	-	√	优化处理时序信息
IRBM^[46]	增加能量惩罚项	√	-	-	无需设置隐藏层节点
DIRBM^[47]	增加标签信息联合分布	√	-	-	无需隐层超参数的自主分类器
3RBM^[48]	删除隐藏层冗余信息	-	√	-	大幅度优化模型结构
F3RBM^[49]	引入模糊隶属度函数	-	√	-	增强模型鲁棒性
giRBM^[50]	引入图的正则化方案进行编码	-	-	√	使用在文档建模领域
PCRBM^[51]	隐藏层特征的一致性规则	-	-	√	可以进行多视图学习领域学习
PDRBM^[52]	学习多视图领域的数据独有信息	-	-	√	优化多视图学习
mgRBM^[53]	学习领域信息	-	-	√	进一步优化多视图学习
E-RBM^[54]	基于能量判断隐藏层节点重要程度	-	√	-	基于能量简化模型结构
SCRBM^[55]	在时序RBM中添加标签信息	√	-	√	可以处理时序数据并分类
CVRBM^[56]	添加对称虚数部神经网络结构	√	-	√	有效处理复值数据

单元名称	激活函数	高斯近似	条件概率
Sigmoid	$\left(1+\mathrm{e}^{-\eta}\right)^{-1}$	-	$\exp (\eta h-\log (1+\exp (\eta)))$
Noisy Tanh	$\left(1+\mathrm{e}^{-\eta}\right)^{-1}-1 / 2$	$N(f(\eta), (f(\eta)-1 / 2)(f(\eta)+1 / 2))$	$\exp (\eta h-\log (1+\exp (\eta))+\operatorname{ent}(h)+g(h))$
ArcSinh	$\log \left(\eta+\sqrt{1+\eta^2}\right)$	$N\left(\sinh (\eta), \left(\sqrt{1+\eta^2}\right)^{-1}\right)$	$\exp \left(\eta h-\cos h+\sqrt{1+\eta^2}-\eta \sin ^{-1}(\eta)+g(h)\right)$
SymSqrt	$\operatorname{sign}(\eta) \sqrt{\|\eta\|}$	$N(f(\eta), \sqrt{\|\eta\|} / 2)$	$\exp \left(\eta h-\|h\|^3 / 3-2\left(\eta^2\right)^{2 / 3} / 3+g(h)\right)$
Linear	η	$N(\eta, 1)$	$\exp \left(\eta h-\eta^2 / 2-h^2 / 2-\log (\sqrt{2 \pi})\right)$
Softplus	$\log \left(1+\mathbf{e}^\eta\right)$	$N\left(f(\eta), \left(1+\mathrm{e}^{-\eta}\right)^{-1}\right)$	$\exp \left(\eta h-2 \mathrm{Li}_2\left(-\mathrm{e}^\eta\right)-h \log \left(1-\mathrm{e}^h\right)+y \log \left(\mathrm{e}^\eta-1\right)+g(h)\right)$
ReLU	$\max (0, \eta)$	$N(f(\eta), \Pi(\eta))$	-
ReQU	$\max (0, \eta\|\eta\|)$	$N(f(\eta), \Pi(\eta) \eta)$	-
SymQU	$\eta\|\eta\|$	$N(\eta\|\eta\|, \|\eta\|)$	$\exp \left(\eta h-\|h\|^3 / 3-2\left(\eta^2\right)^{3 / 3} / 3+g(h)\right)$
Exponential	$\mathrm{e}^\eta$	$N\left(\mathrm{e}^\eta, \mathrm{e}^\eta\right)$	$\exp \left(\eta h-\mathrm{e}^\eta-h(\log (y)-1)+g(h)\right)$
Sinh	$1 / 2\left(\mathrm{e}^\eta-\mathrm{e}^{-\eta}\right)$	$N(\sinh (\eta), \cosh (\eta))$	$\exp \left(\eta h-\cos h+\sqrt{1+\eta^2}-\eta \sin ^{-1} \eta+g(h)\right)$
Poisson	$\mathrm{e}^\eta$	-	$\exp \left(\eta h-\mathrm{e}^\eta-y!\right)$

模型	融合的生成模型	生成性	判别性	优势
HBM^[81-82]	HN	√	-	学习检索性能提升
FDBN^[83]	DBN	√	√	高维数据处理能力加强
CRBM-DNN^[84]	DNN	√	-	添加先验知识, 稀疏昂贵缺失样本学习能力优化
RBM-RNN^[85]	RNN	√	√	恶意流量检测性能加强
RB-ELM^[86]	ELM	-	√	结构简化, 故障检测性能提升
RBM-ANN^[87]	ANN	√	-	图像重构性能优化

Review of research on restricted Boltzmann machine and its variants

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 25

References 124

Related Articles 15

Recommended Articles

Metrics

Comments

[1]	Xiantao SUN, Wangyang JIANG, Wenjie CHEN, Weihai CHEN, Yali ZHI. Object grasp pose detection based on the region of interest [J]. Systems Engineering and Electronics, 2024, 46(6): 1867-1877.
[2]	Pengfei YANG, Ling HE, Qian WANG, Ruidi WANG, Mingzhi ZHANG. Interference identification based on mixed signal multidomain feature and Transformer framework [J]. Systems Engineering and Electronics, 2024, 46(6): 2138-2145.
[3]	Xuemei CHEN, Zhiheng LIU, Suiping ZHOU, Hang YU, Yanming LIU. Road extraction from high-resolution remote sensing images based on HRNet [J]. Systems Engineering and Electronics, 2024, 46(4): 1167-1173.
[4]	Tianwen ZHANG, Xiaoling ZHANG, Zikang SHAO, Tianjiao ZENG. Mask attention interaction for SAR ship instance segmentation [J]. Systems Engineering and Electronics, 2024, 46(3): 831-838.
[5]	Degui YANG, Daofeng XU. Human behavior recognition method of IR-UWB through wall radar based on time-frequency domain feature fusion [J]. Systems Engineering and Electronics, 2024, 46(3): 849-858.
[6]	Duanyang SHI, Qiang LIN, Bing HU, Xiaoshuai DU. Target detection method of primary surveillance radar based on YOLO [J]. Systems Engineering and Electronics, 2024, 46(1): 143-151.
[7]	Xiaofeng ZHAO, Jiahui NIU, Chuntong LIU, Yuting XIA. Hyperspectral image classification based on hybrid convolution with three-dimensional attention mechanism [J]. Systems Engineering and Electronics, 2023, 45(9): 2673-2680.
[8]	Shiyan SUN, Gang ZHANG, Weige LIANG, Bo SHE, Fuqing TIAN. Construction method of rolling bearing health indicator based on enhanced restricted Boltzmann machine [J]. Systems Engineering and Electronics, 2023, 45(9): 2979-2985.
[9]	Meng WANG, Bing ZHU. Application of uncertainty modeling in 2D and 3D object detection [J]. Systems Engineering and Electronics, 2023, 45(8): 2370-2376.
[10]	Wei FANG, Jingwen LIANG, Hengyang LU. Genetic programming algorithm based on cluster tournament and parent matching [J]. Systems Engineering and Electronics, 2023, 45(8): 2405-2414.
[11]	Kai SHAO, Ziqun DU, Guangyu WANG. CSI feedback method for dynamically adjusting compression rate based on model pruning [J]. Systems Engineering and Electronics, 2023, 45(8): 2615-2622.
[12]	Husheng WANG, Baixiao CHEN, Qingzhi YE. Research on anti-chaff jamming method based on measured data [J]. Systems Engineering and Electronics, 2023, 45(7): 2010-2021.
[13]	Fan YANG, Ping MA, Wei LI, Ming YANG. Intelligent ranking evaluation method of simulation models based on siamese network [J]. Systems Engineering and Electronics, 2023, 45(7): 2060-2068.
[14]	Tianshu CUI, Dong WANG, Zhen HUANG. Automatic modulation classification based on lightweight network for space cognitive communication [J]. Systems Engineering and Electronics, 2023, 45(7): 2220-2226.
[15]	Yu JIANG, Qi YUAN, Zhitao HU, Weiwei WU, Xin GU. Airport arrival and departure delay time prediction based on meteorological factors [J]. Systems Engineering and Electronics, 2023, 45(6): 1722-1731.