超维计算概念、应用及研究进展

doi:10.12305/j.issn.1001-506X.2023.07.04

Abstract

Abstract:

Hyperdimensional computing (HDC) is an emerging brain-inspired cognitive model. It uses high-dimensional random holographic distributed representations of different entities as processing objects and has many advantages such as low computational cost, superfast learning process, high hardware-friendliness, strong robustness, independent of data volume, and excellent model interpretability. HDC is a promising method in many fields, such as target recognition, signal processing, multi-task learning, information fusion, and smart decision-making. In recent years, HDC has received increasing attention and shown great development potential, which provides researchers with a new choice. This paper introduces the development history, basic principles, and the specific model architecture of HDC, presents some typical application examples, and discusses the existing deficiencies and possible future directions of HDC.

Key words: hyperdimensional computing (HDC), category recognition, distributed representation, machine learning, research progress

CLC Number:

TP181

Wenbo LIU, Yirong YAO, Gong ZHANG, Wen HU. Concept, application, and research progress of hyperdimensional computing[J]. Systems Engineering and Electronics, 2023, 45(7): 1938-1956.

Figures/Tables 17

Fig.1

Fig.2

Fig.3

Fig.4

Fig.5

Fig.6

Fig.7

Fig.8

Fig.9

Fig.10

Fig.11

Fig.12

Fig.13

Fig.14

Fig.15

Fig.16

Fig.17

References 81

1	ROSENBLATT F . The perceptron: a probabilistic model for information storage and organization in the brain[J]. Psychological Review, 1958, 65 (6): 386. doi: 10.1037/h0042519
2	RUMELHART D E , MCCLELLAND J L . Parallel distributed processing: explorations in the microstructure of cognition[M]. Cambridge: MIT Press, 1986.
3	ANDERSON J A . A simple neural network generating an inte-ractive memory[J]. Mathematical Biosciences, 1972, 14 (3/4): 197- 220.
4	KOHONEN T . Correlation matrix memories[J]. IEEE Trans.on Computers, 1972, 100 (4): 353- 359.
5	FUKUSHIMA K . Neocognitron: a self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position[J]. Biological Cybernetics, 1980, 36, 193- 202. doi: 10.1007/BF00344251
6	OLSHAUSEN B A , FIELD D J . Emergence of simple-cell receptive field properties by learning a sparse code for natural images[J]. Nature, 1996, 381 (6583): 607- 609. doi: 10.1038/381607a0
7	BELL A J , SEJNOWSKI T J . An information-maximization approach to blind separation and blind deconvolution[J]. Neural Computation, 1995, 7 (6): 1129- 1159. doi: 10.1162/neco.1995.7.6.1129
8	KRIZHEVSKY A, SUTSKEVER I, HINTON G E. Imagenet classification with deep convolutional neural networks[C]//Proc. of the Conference on Advances in Neural Information Processing Systems, 2012, 25: 1097-1105.
9	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778.
10	SUTSKEVER I, VINYALS O, LE Q V. Sequence to sequence learning with neural networks[C]//Proc. of the Conference on Advances in Neural Information Processing Systems, 2014: 3104-3112.
11	HINTON G E , DENG L , YU D , et al. Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups[J]. IEEE Signal Processing Magazine, 2012, 29 (6): 82- 97. doi: 10.1109/MSP.2012.2205597
12	LI F F , FERGUS R , PERONA P . One-shot learning of object categories[J]. IEEE Trans.on Pattern Analysis and Machine Intelligence, 2006, 28 (4): 594- 611. doi: 10.1109/TPAMI.2006.79
13	LIU Z, LI J G, SHEN Z Q, et al. Learning efficient convolutional networks through network slimming[C]//Proc. of the IEEE International Conference on Computer Vision, 2017: 2736-2744.
14	ZEILER M D, FERGUS R. Visualizing and understanding convolutional networks[C]//Proc. of the European Conference on Computer Vision, 2014: 818-833.
15	ZHANG Y , TINˇO P , LEONARDIS A , et al. A survey on neural network interpretability[J]. IEEE Trans.on Emerging Topics in Computational Intelligence, 2021, 5 (5): 726- 742. doi: 10.1109/TETCI.2021.3100641
16	THOMAS A , DASGUPTA S , ROSING T . Theoretical foundations of hyperdimensional computing[J]. Journal of Artificial Intelligence Research, 2021, 72, 215- 249. doi: 10.1613/jair.1.12664
17	KANERVA P . Sparse distributed memory[M]. Cambridge: MIT Press, 1988.
18	KANERVA P . Hyperdimensional computing: an introduction to computing in distributed representation with high-dimensional random vectors[J]. Cognitive Computation, 2009, 1 (2): 139- 159. doi: 10.1007/s12559-009-9009-8
19	MASSE N Y , TURNER G C , JEFFERIS G S X E . Olfactory information processing in Drosophila[J]. Current Biology, 2009, 19 (16): R700- R713. doi: 10.1016/j.cub.2009.06.026
20	CARON S J C , RUTA V , ABBOTT L F , et al. Random convergence of olfactory inputs in the Drosophila mushroom body[J]. Nature, 2013, 497 (7447): 113- 117. doi: 10.1038/nature12063
21	SQUIRE L R . Memory and the hippocampus: a synthesis from findings with rats, monkeys, and humans[J]. Psychological Review, 1992, 99 (2): 195- 231. doi: 10.1037/0033-295X.99.2.195
22	RAHIMI A, KANERVA P, RABAEY J M. A robust and energy-efficient classifier using brain-inspired hyperdimensional computing[C]//Proc. of the International Symposium on Low Power Electronics and Design, 2016: 64-69.
23	ZHANG S, WANG R, ZHANG J J, et al. Assessing robustness of hyperdimensional computing against errors in associative memory[C]//Proc. of the IEEE 32nd International Conference on Application-specific Systems, Architectures and Processors, 2021: 211-217.
24	HERSCHE M , LIPPUNER S , KORB M , et al. Near-channel classifier: symbiotic communication and classification in high-dimensional space[J]. Brain Informatics, 2021, 8 (1): 16. doi: 10.1186/s40708-021-00138-0
25	KARUNARATNE G , SCHMUCK M , LE GALLO M , et al. Robust high-dimensional memory-augmented neural networks[J]. Nature Communications, 2021, 12 (1): 2468. doi: 10.1038/s41467-021-22364-0
26	YAO Y R , LIU W B , ZHANG G , et al. Radar-based human activity recognition using hyperdimensional computing[J]. IEEE Trans.on Microwave Theory and Techniques, 2022, 70 (3): 1605- 1619. doi: 10.1109/TMTT.2021.3134992
27	GE L L , PARHI K K . Classification using hyperdimensional computing: a review[J]. IEEE Circuits and Systems Magazine, 2020, 20 (2): 30- 47. doi: 10.1109/MCAS.2020.2988388
28	HASSAN E , HALAWANI Y , MOHAMMAD B , et al. Hyper-dimensional computing challenges and opportunities for AI applications[J]. IEEE Access, 2021, 10, 97651- 97664.
29	IMANI M, KONG D, RAHIMI A, et al. VoiceHD: hyperdimensional computing for efficient speech recognition[C]//Proc. of the IEEE International Conference on Rebooting Computing, 2017.
30	JOSHI A, HALSETH J T, KANERVA P. Language geometry using random indexing[C]//Proc. of the International Symposium on Quantum Interaction, 2016: 265-274.
31	IMANI M , HWANG J , ROSING T , et al. Low-power sparse hyperdimensional encoder for language recognition[J]. IEEE Design and Test, 2017, 34 (6): 94- 101. doi: 10.1109/MDAT.2017.2740839
32	MA D N, GUO J M, JIANG Y, et al. HDTest: differential fuzz testing of brain-inspired hyperdimensional computing[C]//Proc. of the ACM/IEEE Design Automation Conference, 2021: 391-396.
33	YAO Y R , LIU W B , ZHANG G , et al. Fast SAR image recog-nition via hyperdimensional computing using monogenic mapping[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 19, 4508205.
34	RAHIMI A , KANERVA P , BENINI L , et al. Efficient biosignal processing using hyperdimensional computing: network templates for combined learning and classification of ExG signals[J]. Proceedings of the IEEE, 2018, 107 (1): 123- 143.
35	RAHIMI A, BENATTI S, KANERVA P, et al. Hyperdimensional biosignal processing: a case study for EMG-based hand gesture recognition[C]//Proc. of the IEEE International Conference on Rebooting Computing, 2016.
36	BURRELLO A , SCHINDLER K , BENINI L , et al. Hyperdimensional computing with local binary patterns: one-shot learning of seizure onset and identification of ictogenic brain regions using short-time iEEG recordings[J]. IEEE Trans.on Biome-dical Engineering, 2019, 67 (2): 601- 613.
37	KIM Y, IMANI M, MOSHIRI N, et al. GenieHD: efficient DNA pattern matching accelerator using hyperdimensional computing[C]//Proc. of the Design, Automation and Test in Europe Conference and Exhibition, 2020: 115-120.
38	CHANG C Y , CHUANG Y , CHANG E J , et al. MulTa-HDC: a multi-task learning framework for hyperdimensional computing[J]. IEEE Trans.on Computers, 2021, 70 (8): 1269- 1284. doi: 10.1109/TC.2021.3073409
39	RÄSÄNEN O J , SAARINEN J P . Sequence prediction with sparse distributed hyperdimensional coding applied to the analysis of mobile phone use patterns[J]. IEEE Trans.on Neural Networks and Learning Systems, 2015, 27 (9): 1878- 1889.
40	RASANEN O , KAKOUROS S . Modeling dependencies in multiple parallel data streams with hyperdimensional computing[J]. IEEE Signal Processing Letters, 2014, 21 (7): 899- 903. doi: 10.1109/LSP.2014.2320573
41	SCHLEGEL K, MIRUS F, NEUBERT P, et al. Multivariate time series analysis for driving style classification using neural networks and hyperdimensional computing[C]//Proc. of the IEEE Intelligent Vehicles Symposium, 2021: 602-609.
42	RAHIMI A, KANERVA P, MILLAN J R, et al. Hyperdimensional computing for noninvasive brain-computer interfaces: blind and one-shot classification of EEG error-related potentials[C]//Proc. of the ACM/EAI International Conference on Bio-inspired Information and Communications Technologies, 2017: 19-26.
43	IMANI M, MORRIS J, BOSCH S, et al. AdaptHD: adaptive efficient training for brain-inspired hyperdimensional computing[C]//Proc. of the IEEE Biomedical Circuits and Systems Conference, 2019.
44	RAHIMI A , DATTA S , KLEYKO D , et al. High-dimensional computing as a nanoscalable paradigm[J]. IEEE Trans.on Circuits and Systems I: Regular Papers, 2017, 64 (9): 2508- 2521. doi: 10.1109/TCSI.2017.2705051
45	IMANI M, MESSERLY J, WU F, et al. A binary learning framework for hyperdimensional computing[C]//Proc. of the Design, Automation and Test in Europe Conference & Exhibition, 2019: 126-131.
46	SCHMUCK M , BENINI L , RAHIMI A . Hardware optimizations of dense binary hyperdimensional computing: remateria-lization of hypervectors, binarized bundling, and combinational associative memory[J]. ACM Journal on Emerging Technologies in Computing Systems, 2019, 15 (4): 32.
47	SALAMAT S , IMANI M , ROSING T . Accelerating hyperdimensional computing on FPGAs by exploiting computational reuse[J]. IEEE Trans.on Computers, 2020, 69 (8): 1159- 1171. doi: 10.1109/TC.2020.2992662
48	LI H, WU T F, RAHIMI A, et al. Hyperdimensional computing with 3D VRRAM in-memory kernels: device-architecture co-design for energy-efficient, error-resilient language recognition[C]//Proc. of the IEEE International Electron Devices Meeting, 2016.
49	WU T F, LI H, HUANG P C, et al. Brain-inspired computing exploiting carbon nanotube FETs and resistive RAM: hyperdimensional computing case study[C]//Proc. of the IEEE International Solid-State Circuits Conference, 2018: 492-494.
50	KARUNARATNE G , LE-GALLO M , CHERUBINI G , et al. In-memory hyperdimensional computing[J]. Nature Electro-nics, 2020, 3 (6): 327- 337. doi: 10.1038/s41928-020-0410-3
51	HOPFIELD J J . Neural networks and physical systems with emergent collective computational abilities[J]. Proceedings of the National Academy of Sciences, 1982, 79 (8): 2554- 2558. doi: 10.1073/pnas.79.8.2554
52	ACKLEY D H , HINTON G E , SEJNOWSKI T J . A learning algorithm for Boltzmann machines[J]. Cognitive Science, 1985, 9 (1): 147- 169. doi: 10.1207/s15516709cog0901_7
53	HINTON G E , OSINDERO S , TEH Y W . A fast learning algorithm for deep belief nets[J]. Neural Computation, 2006, 18 (7): 1527- 1554. doi: 10.1162/neco.2006.18.7.1527
54	HINTON G E , ANDERSON J A . Parallel models of associative memory: updated edition[M]. New York: Psychology Press, 2014.
55	MINSKY M , PAPERT S . Perceptrons[M]. Cambridge: MIT Press, 1969.
56	WERBOS P J . Backpropagation through time: what it does and how to do it[J]. Proceedings of the IEEE, 1990, 78 (10): 1550- 1560. doi: 10.1109/5.58337
57	RUMELHART D E , HINTON G E , WILLIAMS R J . Learning representations by back-propagating errors[J]. Nature, 1986, 323 (6088): 533- 536. doi: 10.1038/323533a0
58	LECUN Y , BOSER B , DENKER J S , et al. Backpropagation applied to handwritten zip code recognition[J]. Neural Computation, 1989, 1 (4): 541- 551. doi: 10.1162/neco.1989.1.4.541
59	HINTON G E . Mapping part-whole hierarchies into connectionist networks[J]. Artificial Intelligence, 1990, 46 (1/2): 47- 75.
60	SMOLENSKY P . Tensor product variable binding and the representation of symbolic structures in connectionist systems[J]. Artificial Intelligence, 1990, 46 (1/2): 159- 216.
61	PLATE T A . Holographic reduced representations[J]. IEEE Trans.on Neural Networks, 1995, 6 (3): 623- 641. doi: 10.1109/72.377968
62	KANERVA P. Binary spatter-coding of ordered k-tuples[C]//Proc. of the International Conference on Artificial Neural Networks, 1996: 869-873.
63	GAYLER R W. Multiplicative binding representation operators and analogy[C]//Proc. of the Workshop on Advances in Analogy Research: Integration of Theory and Data from the Cognitive Computational and Neural Sciences, 1998.
64	PLATE T A . Holographic reduced representations: distributed representation for cognitive structures[M]. Stanford: CSLI, 2003.
65	LEVY S D, GAYLER R. Vector symbolic architectures: a new building material for artificial general intelligence[C]//Proc. of the Conference on Artificial General Intelligence, 2008: 414-418.
66	IMANI M, HUANG C, KONG D, et al. Hierarchical hyperdimensional computing for energy efficient classification[C]//Proc. of the ACM/ESDA/IEEE Design Automation Confe-rence, 2018.
67	QUASTHOFF U, RICHTER M, BIEMANN C. Corpus portal for search in monolingual corpora[C]//Proc. of the International Conference on Language Resources and Evaluation, 2006: 1799-1802.
68	KOEHN P. Europarl: a parallel corpus for statistical machine translation[C]//Proc. of the Machine Translation Summit X, 2005: 79-86.
69	KLEYKO D , OSIPOV E , SENIOR A , et al. Holographic graph neuron: a bioinspired architecture for pattern processing[J]. IEEE Trans.on Neural Networks and Learning Systems, 2016, 28 (6): 1250- 1262.
70	MANABAT A X, MARCELO C R, QUINQUITO A L, et al. Performance analysis of hyperdimensional computing for cha-racter recognition[C]//Proc. of the International Symposium on Multimedia and Communication Technology, 2019.
71	IMANI M, KIM Y, WORLEY T, et al. HDCluster: an accurate clustering using brain-inspired high-dimensional computing[C]//Proc. of the Design, Automation and Test in Europe Conference & Exhibition, 2019: 1591-1594.
72	KLEYKO D , RAHIMI A , RACHKOVSKIJ D A , et al. Classification and recall with binary hyperdimensional computing: tradeoffs in choice of density and mapping characteristics[J]. IEEE Trans.on Neural Networks and Learning Systems, 2018, 29 (12): 5880- 5898. doi: 10.1109/TNNLS.2018.2814400
73	IMANI M, SALAMAT S, KHALEGHI B, et al. SparseHD: algorithm-hardware co-optimization for efficient high-dimensional computing[C]//Proc. of the IEEE 27th Annual International Symposium on Field-Programmable Custom Computing Machines, 2019: 190-198.
74	MITROKHIN A , SUTOR P , SUMMERS-STAY D , et al. Symbolic representation and learning with hyperdimensional computing[J]. Frontiers in Robotics and AI, 2020, 7, 63. doi: 10.3389/frobt.2020.00063
75	ALONSO P, SHRIDHAR K, KLEYKO D, et al. HyperEmbed: tradeoffs between resources and performance in NLP tasks with hyperdimensional computing enabled embedding of n-gram statistics[C]//Proc. of the International Joint Conference on Neural Networks, 2021.
76	KLEYKO D , KHEFFACHE M , FRADY E P , et al. Density encoding enables resource-efficient randomly connected neural networks[J]. IEEE Trans.on Neural Networks and Learning Systems, 2020, 32 (8): 3777- 3783.
77	MORRIS J, LUI H W, STEWART K, et al. HyperSpike: hyperdimensional computing for more efficient and robust spiking neural networks[C]//Proc. of the 2022 Design, Automation & Test in Europe Conference & Exhibition, 2022: 664-669.
78	KLEYKO D , FRADY E P , KHEFFACHE M , et al. Integer echo state networks: efficient reservoir computing for digital hardware[J]. IEEE Trans.on Neural Networks and Learning Systems, 2022, 33 (4): 1688- 1701. doi: 10.1109/TNNLS.2020.3043309
79	HSU P K, YU S. In-memory 3D NAND flash hyperdimensional computing engine for energy-efficient SARS-CoV-2 genome sequencing[C]//Proc. of the IEEE International Memory Workshop, 2022.
80	DUBREUIL T, AMARI P, BARRAUD S, et al. A novel 3D 1T1R RRAM architecture for memory-centric hyperdimensional computing[C]//Proc. of the IEEE International Memory Workshop, 2022.
81	DATTA S , ANTONIO R A G , ISON A R S , et al. A programmable hyper-dimensional processor architecture for human-centric IoT[J]. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 2019, 9 (3): 439- 452. doi: 10.1109/JETCAS.2019.2935464

[1]	Husheng WANG, Baixiao CHEN, Qingzhi YE. Research on anti-chaff jamming method based on measured data [J]. Systems Engineering and Electronics, 2023, 45(7): 2010-2021.
[2]	Chunzheng WANG, Minghua HU, Lei YANG, Zheng ZHAO. Review on air traffic delay prediction [J]. Systems Engineering and Electronics, 2022, 44(3): 863-874.
[3]	Xinyu ZHANG, Yuan LIU, Jianing SONG. Short-term orbit prediction based on LSTM neural network [J]. Systems Engineering and Electronics, 2022, 44(3): 939-947.
[4]	Fan ZHAO, Hu JIN. Communication jamming waveform generation technology based on GAN [J]. Systems Engineering and Electronics, 2021, 43(4): 1080-1088.
[5]	Bo SONG, Wei YE, Xianghui MENG. Review of multi-agent reinforcement learning based dynamic spectrum allocation method [J]. Systems Engineering and Electronics, 2021, 43(11): 3338-3351.
[6]	Fang PENG, Jun WU, Shuai WANG, Jianjun XIANG. Adaptive beamforming algorithm for airborne early warning radar based on SVRGD [J]. Systems Engineering and Electronics, 2021, 43(1): 83-90.
[7]	ZHANG Jing, HE You, PENG Yingning, LI Gang. Technical features demonstration of the next generation fighter [J]. Systems Engineering and Electronics, 2018, 40(8): 1754-1759.
[8]	ZHUANSUN Shaoshuai, YANG Junan, LIU Hui, HUANG Keju. Jamming strategy learning based on positive reinforcement learning and orthogonal decomposition [J]. Systems Engineering and Electronics, 2018, 40(3): 518-525.
[9]	WANG Yi, SUN Jin-biao, XIAO Ming-qing, LUO Ji-xun. Research of dynamic weapon-target assignment problem based on type-2 interval fuzzy K-nearest neighbors classifier [J]. Systems Engineering and Electronics, 2016, 38(6): 1314-1319.
[10]	GU Bin-jie, PAN Feng. Improved v solution path for v-support vector regression [J]. Systems Engineering and Electronics, 2016, 38(1): 205-214.

Concept, application, and research progress of hyperdimensional computing

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 17

References 81

Related Articles 10

Recommended Articles

Metrics

Comments