基于Policy Gradient的机械臂运动跟踪

doi:10.12305/j.issn.1001-506X.2021.09.29

系统工程与电子技术 ›› 2021, Vol. 43 ›› Issue (9): 2605-2611.doi: 10.12305/j.issn.1001-506X.2021.09.29

基于Policy Gradient的机械臂运动跟踪

韩霖骁*, 胡剑波, 宋仕元, 王应洋, 贺子厚, 张鹏

空军工程大学装备管理与无人机工程学院, 陕西西安 710051

收稿日期:2020-10-30 出版日期:2021-08-20 发布日期:2021-08-26
通讯作者: 韩霖骁
作者简介:韩霖骁(1997—), 男, 硕士研究生, 主要研究方向为智能控制与无人作战|胡剑波(1965—), 男, 教授, 博士, 主要研究方向为滑模控制、智能控制与无人作战|宋仕元(1996—), 男, 硕士研究生, 主要研究方向为智能控制与无人作战|王应洋(1990—), 男, 讲师，博士, 主要研究方向为智能控制与无人作战|贺子厚(1995—), 男, 博士研究生, 主要研究方向为智能控制与无人作战|张鹏(1979—), 男, 副教授, 博士, 主要研究方向为智能控制与无人作战
基金资助:
工业控制技术国家重点实验室开放课题(ICT20063)

Parameter tuning of manipulator motion tracking controller based on Policy Gradient

Linxiao HAN*, Jianbo HU, Shiyuan SONG, Yingyang WANG, Zihou HE, Peng ZHANG

Equipment Management and Unmanned Aerial Vehicle Engineering College, Airforce and Engineering University, Xi'an 710051, China

Received:2020-10-30 Online:2021-08-20 Published:2021-08-26
Contact: Linxiao HAN

摘要/Abstract

摘要：

针对机械臂运动跟踪控制器的参数自整定问题, 设计了一种基于强化学习Policy Gradient法的参数整定器。首先, 介绍了机械臂的一种混合动力学模型, 根据该系统模型进行了比例-微分(proportional-derivative, PD)控制器设计和李雅普诺夫稳定性证明, 并由此给出了参数矩阵的范围。其次, 设计了基于Policy Gradient的参数整定器, 通过引入积分器的方法对其进行改进, 使其控制下的参数行为连续化以进一步提高PD控制器的控制效果。最后, 以二阶机械臂系统为例进行了仿真验证。实验数据证明了该参数整定器的有效性和可行性, 并能有效提升系统的动态性能。

关键词: 机械臂, 运动跟踪, Policy Gradient, 参数整定, 比例-微分控制

Abstract:

Aiming at the parameter self-tuning problem of the manipulator motion tracking controller, a parameter tuning device based on reinforcement learning Policy Gradient method is designed. Firstly, a hybrid dynamic model of manipulator is introduced. According to the system model, the design of proportional-derivative (PD) controller and the stability of Lyapunov are carried out, and the range of parameter matrix is given. Secondly, the parameter tuning device based on Policy Gradient is designed. The integrator is introduced to improve the parameter tuning of gradient, so as to further improve the control effect of PD controller. Finally, a second-order manipulator system is taken as an example to verify the effectiveness and feasibility of the parameter tuning device, which effectively improves the dynamic performance of the proposed system.

Key words: manipulator, motion tracking, Policy Gradient, parameter tuning, proportional-derivative (PD) control

中图分类号:

O231.2

韩霖骁, 胡剑波, 宋仕元, 王应洋, 贺子厚, 张鹏. 基于Policy Gradient的机械臂运动跟踪[J]. 系统工程与电子技术, 2021, 43(9): 2605-2611.

Linxiao HAN, Jianbo HU, Shiyuan SONG, Yingyang WANG, Zihou HE, Peng ZHANG. Parameter tuning of manipulator motion tracking controller based on Policy Gradient[J]. Systems Engineering and Electronics, 2021, 43(9): 2605-2611.

图/表 7

图1

图2

图3

图4

图5

图6

图7

参考文献 31

1	李小华, 杨瑞芳, 刘辉. 一类机械臂系统自适应有限时间有界H∞跟踪控制[J]. 控制理论与应用, 2021, 38 (1): 147- 156.
	LI X H , YANG R F , LIU H . Adaptive finite-time bounded-H∞ tracking control for a class of manipulator system[J]. Control Theory & Applications, 2021, 38 (1): 147- 156.
2	WAI R J , YANG Z W . Adaptive fuzzy neural network control design via a T-S fuzzy model for a robot manipulator including actuator dynamics[J]. IEEE Trans.on Systems, Man, and Cybernetics Part B, 2008, 38 (5): 1326- 1346. doi: 10.1109/TSMCB.2008.925749
3	PILTAN F , SULAIMAN N , ALLAHDADI S , et al. Position control of robot manipulator: design a novel SISO adaptive sliding mode fuzzy PD fuzzy sliding mode control[J]. International Journal of Artificial Intelligence & Expert Systems, 2011, 2 (5): 208- 228.
4	NIKDEL N , BADAMCHIZADEH M A , AZIMIRAD V , et al. Adaptive backstepping control for an n-degree of freedom robotic manipulator based on combined state augmentation[J]. Robotics and Computer Integrated Manufacturing, 2017, 44, 129- 143. doi: 10.1016/j.rcim.2016.08.007
5	PERRUSQUIA A , FLORES J A , TORRES C R . A novel tuning method of PD with gravity compensation controller for robot manipulators[J]. IEEE Access, 2020, 8, 114773- 114783. doi: 10.1109/ACCESS.2020.3003842
6	SU Y X , SUN D , REN L , et al. Integration of saturated PI synchronous control and PD feedback for control of parallel manipulator[J]. IEEE Trans.on Robotics, 2006, 22 (1): 202- 207. doi: 10.1109/TRO.2005.858852
7	NPRPUZI A, KOCH C R. robotic manipulator control using PD-type fuzzy iterative learning control[C]//Proc. of the IEEE Canadian Conference of Electrical and Computer Engineering, 2019.
8	ZHANG H M, GAO X Y, XU G Y. Research on improved PD control of flexible manipulator[C]//Proc. of the Chinese Control and Decision Conference, 2019: 5053-5057.
9	DONG J , HE B , ZHANG C H , et al. Open-closed-loop PD itera-tive learning control corrected with the angular relationship of output vectors for a flexible manipulator[J]. IEEE Access, 2019, 7, 167815- 167822. doi: 10.1109/ACCESS.2019.2930559
10	LIU Y H , SUN D . Stabilizing a flexible beam handled by two manipulators via PD feedback[J]. IEEE Trans.on Automatic Control, 2000, 45 (11): 2159- 2164. doi: 10.1109/9.887656
11	ZHANG C Y . PD plus dynamic pressure feedback control for a direct drive stewart manipulator[J]. Energies, 2020, 13 (5): 1125- 1131. doi: 10.3390/en13051125
12	BENCHAREF S, BOUBERTAKH H. Optimal tuning of a PD control by bat algorithm to stabilize a quadrotor[C]//Proc. of the IEEE 8th International Conference on Modelling, Identification and Control, 2017: 938-942.
13	KHATOON S , NASIRUDDIN I , SHAHID M . Design and simulation of a hybrid PD-ANFIS controller for attitude tracking control of a quadrotor UAV[J]. Arabian Journal for Science and Engineering, 2017, 42, 5211- 5229. doi: 10.1007/s13369-017-2586-z
14	ARROSIDA H, EFFENDI R, AGUSTINAH T, et al. Design of decoupling and nonlinear PD controller for cruise control of a quadrotor[C]//Proc. of the International Seminar on Intelligent Technology and its Applications, 2015: 57-62.
15	LU S T , ZHAO J D . Research on tracking control of circular tra-jectory of robot based on the variable integral sliding mode PD control algorithm[J]. IEEE Acess, 2020, 8, 204194- 204202. doi: 10.1109/ACCESS.2020.3027772
16	KOLATHAYA S . Local stability of PD controlled bipedal walking robots[J]. Automatica, 2020, 114, 108841. doi: 10.1016/j.automatica.2020.108841
17	李洋. PID参数自整定器设计与实现[D]. 黑龙江: 黑龙江大学, 2014.
	LI Y. Design and realization of PID parameter self-tuning device[D]. Heilongjiang: Heilongjiang University, 2014.
18	VERMA B , PADHY P K . Robust fine tuning of optimal PID controller with guaranteed robustness[J]. IEEE Trans.on Industrial Electronics, 2020, 67 (6): 4911- 4920. doi: 10.1109/TIE.2019.2924603
19	QI Z , SHI Q , ZHANG H . Tuning of digital PID controllers using particle swarm optimization algorithm for a CAN-based DC motor subject to stochastic delays[J]. IEEE Trans.on Industrial Electronics, 2020, 67 (7): 5637- 5646. doi: 10.1109/TIE.2019.2934030
20	COATA E B M , SERRA G L O . Self-tuning robust fuzzy controller design based on multi-objective particle swarm optimization adaptation mechanism[J]. Journal of Dynamic Systems Measurement and Control-Transactions of the ASME, 2017, 139 (7): 071009. doi: 10.1115/1.4035758
21	HEKIMOGLU B . Optimal tuning of fractional order PID controller for DC motor speed control via chaotic atom search optimization algorithm[J]. IEEE Access, 2019, 7, 38100- 38114. doi: 10.1109/ACCESS.2019.2905961
22	SU Y P , YU Q M , ZENG L . Parameter self-tuning PID control for greenhouse climate control problem[J]. IEEE Access, 2020, 8, 186157- 186171. doi: 10.1109/ACCESS.2020.3030416
23	HEERTJES M , VERSTAPPEN R . Self-tuning in integral sliding mode control with a Levenberg-Marquardt algorithm[J]. Mechatronics, 2014, 24 (4): 385- 393. doi: 10.1016/j.mechatronics.2013.05.005
24	LI J S , LI W G . On-line PID parameters optimization control for wind power generation system based on genetic algorithm[J]. IEEE Access, 2020, 8, 137094- 137100. doi: 10.1109/ACCESS.2020.3009240
25	YANG B , LIANG G A , PENG J H , et al. Self-adaptive PID controller of microwave drying rotary device tuning on-line by genetic algorithms[J]. Journal of Central South University, 2013, 20 (10): 2685- 2692. doi: 10.1007/s11771-013-1784-4
26	LU L , LIANG S , DAI Y W , et al. Improved quantum bacterial foraging algorithm for tuning parameters of fractional-order PID controller[J]. Journal of Systems Engineering and Electronics, 2018, 29 (1): 166- 175. doi: 10.21629/JSEE.2018.01.17
27	李昭莹. 基于Q学习—变论域模糊PID算法在柴油机调速系统中的设计研究[D]. 哈尔滨: 哈尔滨工程大学, 2016.
	LI Z Y. Research on design of diesel engine speed control system based on Q-learning-variable domain fuzzy PID algorithm[D]. Harbin: Harbin Engineering University, 2016.
28	邓凯. 基于强化学习方法的飞行姿态模拟器控制系统设计[D]. 哈尔滨: 哈尔滨工业大学, 2018.
	DENG K. Design of flight attitude simulator control system based on reinforcement learning method[D]. Harbin: Harbin Institute of Technology, 2018.
29	LIU Y , ZHANG C . Application of dueling DQN and DECGA for parameter estimation in variogram models[J]. IEEE Access, 2020, 8, 38112- 38122. doi: 10.1109/ACCESS.2020.2974926
30	宋仕元, 胡剑波, 王应洋, 等. 滑模控制器参数整定的Actor-Critic学习算法[J]. 电光与控制, 2020, 27 (9): 24- 27, 49.
	SONG S Y , HU J B , WANG Y Y , et al. Actor-critic learning algorithm for parameter tuning of sliding mode controller[J]. Electronics Optics & Control, 2020, 27 (9): 24- 27, 49.
31	刘金琨. 先进PID控制MATLAB仿真[M]. 4版北京: 电子工业出版社, 2004.
	LIU J K . Advanced PID control MATLAB simulation[M]. 4th ed Beijing: Electronic Industry Press, 2004.

[1]	万齐天, 卢宝刚, 赵雅心, 温求遒. 基于深度强化学习的驾驶仪参数快速整定方法[J]. 系统工程与电子技术, 2022, 44(10): 3190-3199.
[2]	谭诗利, 雷虎民, 王鹏飞. 基于正切Sigmoid函数的跟踪微分器[J]. 系统工程与电子技术, 2019, 41(7): 1590-1596.
[3]	黄丽莲，周晓亮，项建弘. 分数阶PID控制器参数的自适应设计[J]. Journal of Systems Engineering and Electronics, 2013, 35(5): 1064-1069.

基于Policy Gradient的机械臂运动跟踪

Parameter tuning of manipulator motion tracking controller based on Policy Gradient

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 7

参考文献 31

相关文章 3

编辑推荐

Metrics

本文评价