有/无人机协同操作界面的最佳交互方式评估

doi:10.12305/j.issn.1001-506X.2025.08.17

系统工程与电子技术 ›› 2025, Vol. 47 ›› Issue (8): 2600-2611.doi: 10.12305/j.issn.1001-506X.2025.08.17

• 系统工程 • 上一篇

有/无人机协同操作界面的最佳交互方式评估

张欣悦¹^,²(), 吴晓莉¹^,²^,*(), 王名珺¹^,², 晏彪¹^,², 武愈涵¹^,²

1. 南京理工大学人机融合与智能交互研究中心，江苏南京 210094
2. 语言信息智能处理及应用工信部重点实验室，江苏南京 210094

收稿日期:2024-07-01 出版日期:2025-08-25 发布日期:2025-09-04
通讯作者: 吴晓莉 E-mail:969264885@qq.com;wuxlhhu@163.com
作者简介:张欣悦（2001—），女，硕士研究生，主要研究方向为多模态交互
王名珺（2000—），女，硕士研究生，主要研究方向为人因与人机交互
晏　彪（1996—），男，博士研究生，主要研究方向为人机交互
武愈涵（2001—），女，硕士研究生，主要研究方向为人机交互
基金资助:
国家自然科学基金（52175469）；江苏省自然科学基金（BK20221490）资助课题

Assessing optimal interaction for a cooperative operation interface of manned/unmanned aerial vehicle

Xinyue ZHANG¹^,²(), Xiaoli WU¹^,²^,*(), Mingjun WANG¹^,², Biao YAN¹^,², Yuhan WU¹^,²

1. Research Centre of Human Cyber Physical Integration and Intelligent Interaction，Nanjing University of Science and Technology，Nanjing 210094，China
2. Key Laboratory of Ministry Industry and Information Technology for Language Information Processing and Application，Nanjing 210094，China

Received:2024-07-01 Online:2025-08-25 Published:2025-09-04
Contact: Xiaoli WU E-mail:969264885@qq.com;wuxlhhu@163.com

摘要/Abstract

摘要：

评估有/无人机协同操作界面按键交互与多模态交互对任务效能和认知负荷的影响，同时为各窗口间操作行为匹配最佳交互方式。实验测定被试使用按键交互和多模态交互（触控手势、语音指令、触控手势语音结合）分别完成三类窗口间操作行为任务的行为指标、眼动指标及主观评价。结果表明，多模态交互的各项数据显著优于按键交互。在不同窗口间操作行为中，切换窗口使用触控手势语音结合交互最优，移动窗口和关闭窗口行为使用触控手势交互最优。实验结果验证多模态交互的优势，同时为其他复杂信息系统界面的窗口交互设计提供参考。

关键词: 有/无人机, 协同操作界面, 多模态交互, 按键交互, 窗口间操作

Abstract:

This paper assess the effects of keystroke interaction and multimodal interaction on task performance and cognitive load with a cooperative operation interface of manned/unmanned aerial vehicle, as well as to match the optimal interactions for different inter-window operating behaviors. The experiment tests the behavioral indicator, eye movement data and subjective evaluation of participants using keystroke interactions and multimodal interactions （touch gestures, voice commands, and touch gesture-voice combinations） for each of the three types of basic inter-window operating behaviors tasks. The results show that multimodal interactions significantly outperform keystroke interactions. Among the different inter-window operating behaviors, the optimal interaction for window switching behavior is touch gesture-voice combined interaction, and the optimal interaction for moving window and closing window behavior is touch gesture interaction. The experimental results validate the advantages of multimodal interaction and provide a reference for the inter-window interaction design of other complex information system interfaces.

Key words: manned/unmanned aerial vehicle, cooperative operation interface, multimodal interaction, keystroke interaction, inter-window operation

中图分类号:

张欣悦, 吴晓莉, 王名珺, 晏彪, 武愈涵. 有/无人机协同操作界面的最佳交互方式评估[J]. 系统工程与电子技术, 2025, 47(8): 2600-2611.

Xinyue ZHANG, Xiaoli WU, Mingjun WANG, Biao YAN, Yuhan WU. Assessing optimal interaction for a cooperative operation interface of manned/unmanned aerial vehicle[J]. Systems Engineering and Electronics, 2025, 47(8): 2600-2611.

图/表 24

图1

图2

图3

图4

图5

图6

表1

图7

图8

图9

图10

表2

图11

表3

表4

图12

图13

图14

图15

表5

图16

表6

表7

图17

参考文献 30

1	HU X X, MA H W, YE Q S, et al. Hierarchical method of task assignment for multiple cooperating UAV teams[J]. Journal of Systems Engineering and Electronics, 2015, 26 (5): 1000- 1009. doi: 10.1109/JSEE.2015.00109
2	ZHU X P, LIU Z, YANG J. Model of collaborative UAV swarm toward coordination and control mechanisms study[J]. Procedia Computer Science, 2015, 51, 493- 502. doi: 10.1016/j.procs.2015.05.274
3	张旭东, 孙智伟, 吴利荣, 等. 未来有人机/无人机智能协同作战顶层概念思考[J]. 无人系统技术, 2021, 4 (2): 62- 68.
	ZHANG X D, SUN Z W, WU L R, et al. Top-level conceptual thinking on future manned/unmanned aircraft intelligent cooperative operations[J]. Unmanned Systems Technology, 2021, 4 (2): 62- 68.
4	吴晓莉, 张蓝, 牛佳然, 等. 航战座舱显控交互研究进展与人机协同发展趋势[J]. 包装工程, 2022, 43 (10): 1- 13.
	WU X L, ZHANG L, NIU J R, et al. Research progress of aerospace warfare cockpit display and control interaction and development trends of human-machine collaboration[J]. Packaging Engineering, 2022, 43 (10): 1- 13.
5	LIM Y X, GARDI A, SABATINI R, et al. Avionics human-machine interfaces and interactions for manned and unmanned aircraft[J]. Progress in Aerospace Sciences, 2018, 102, 1- 46. doi: 10.1016/j.paerosci.2018.05.002
6	XUE H J, ZHANG Q P, ZHANG X Y. Research on the applicability of touchscreens in manned/unmanned aerial vehicle cooperative missions[J]. Sensors, 2022, 22 (21): 8435. doi: 10.3390/s22218435
7	JANSEN C, WENNEMERS A, VOS W, et al. Flytact: a tactile display improves a helicopter pilot’s landing performance in degraded visual environments[C]// Proc. of the International Conference on Human Haptic Sensing and Touch Enabled Computer Applications, 2008: 867–875.
8	陶达, 蔡剑, 张旭, 等. 晃动环境下触摸屏手势操作的可用性研究[J]. 工业工程与管理, 2019, 24 (5): 177- 181.
	TAO D, CAI J, ZHANG X, et al. Usability study of touch screen gesture operations in shaking environments[J]. Industrial Engineering and Management, 2019, 24 (5): 177- 181.
9	汪海波, 薛澄岐, 朱玉婷, 等. 多点触控手势在复杂系统数字界面中的应用优势[J]. 东南大学学报（自然科学版）, 2016, 46 (5): 1002- 1006. doi: 10.3969/j.issn.1001-0505.2016.05.018
	WANG H B, XUE C Q, ZHU Y T, et al. Advantages of multi-touch gestures in digital interfaces for complex systems[J]. Journal of Southeast University （Natural Science Edition）, 2016, 46 (5): 1002- 1006. doi: 10.3969/j.issn.1001-0505.2016.05.018
10	SEABORN K, MIYAKE N P, PENNEFATHER P, et al. Voice in human–agent interaction: a survey[J]. ACM Computing Surveys, 2021, 54 (4): 81.
11	KORSUN O, GLUKHOVA E. Advanced multimodal interfaces design using speech control[C]// Proc. of the 2nd International Conference on High-Speed Transport Development, 2023.
12	王晓颖, 李思凝, 刘泽石, 等. 有人机座舱指控人机交互技术[J]. 飞机设计, 2020, 40 (6): 58- 73.
	WANG X Y, LI S N, LIU Z S, et al. Manned cockpit allegations human-machine interaction technology[J]. Aircraft Design, 2020, 40 (6): 58- 73.
13	DUDEK M, SCHULTE A. Effects of tasking modalities in manned-unmanned teaming missions[C]// Proc. of the AIAA Scitech Forum, 2022.
14	张燕雯, 张泉清. 飞机驾驶舱顶部板触控技术工效学研究[J]. 人类工效学, 2022, 28 (4): 45- 48.
	ZHANG Y W, ZHANG Q Q. Ergonomics of touch control technology on the top panel of an airplane cockpit[J]. Chinese Journal of Ergonomics, 2022, 28 (4): 45- 48.
15	CALHOUN G L, RUFF H A, BEHYMER K J, et al. Evaluation of interface modality for control of multiple unmanned vehicles[C]// Proc. of the International Conference on Engineering Psychology and Cognitive Ergonomics, 2017: 15–34.
16	LEVULIS S J, DELUCIA P R, KIM S Y. Effects of touch, voice, and multimodal input, and task load on multiple-UAV monitoring performance during simulated manned-unmanned teaming in a military helicopter[J]. Journal of the Human Factors and Ergonomics Society, 2018, 60 (8): 1117- 1129.
17	LINDNER S, MUND D, SCHULTE A. How human-autonomy teams change the role of future fighter pilots: an experimental assessment[C]// Proc. of the AIAA SCITECH Forum, 2022.
18	LI W C, LIANG Y H, KOREK W T, et al. Assessments on human-computer interaction using touchscreen as control inputs in flight operations[C]// Proc. of the International Conference on Human-Computer Interaction, 2022: 326–338.
19	HO H F, SU H S, LI W C, et al. Pilots’ latency of first fixation and dwell among regions of interest on the flight deck[C]// Proc. of the 13th International Conference on Engineering Psychology and Cognitive Ergonomics, 2016: 389–396.
20	曹元喆. 飞机驾驶舱光环境对视觉工效的影响研究[D]. 太原: 太原理工大学, 2022.
	CAO Y Z. A study of the effect of aircraft cockpit light environment on visual ergonomics[D]. Taiyuan: Taiyuan University of Technology, 2022.
21	IVOSEVIC J, BUCAK T, ANDRASI P. Effects of interior aircraft noise on pilot performance[J]. Applied Acoustics, 2018, 139, 8- 13. doi: 10.1016/j.apacoust.2018.04.006
22	LIU A, WANG Z, FU S. Ergonomic evaluation of the touch screen in the cockpit under stationary and vibration conditions[C]// Proc. of the International Conference on Human - Computer Interaction, 2022.
23	袁义凡. 近十年认知负荷在界面设计中的应用研究综述[J]. 设计, 2020, 33 (17): 116- 118.
	YUAN Y F. A review of research on the application of cognitive load in interface design in the last decade[J]. Design, 2020, 33 (17): 116- 118.
24	BRAARUD P. Investigating the validity of subjective workload rating （NASA TLX） and subjective situation awareness rating （SART） for cognitively complex human–machine work[J]. International Journal of Industrial Ergonomics, 2021, 86, 103233. doi: 10.1016/j.ergon.2021.103233
25	KRAMER A F. Physiological metrics of mental workload: a review of recent progress[M]. Boca Raton: CRC Press, 2020.
26	姬鸣, 解旭东, 邱燕. 飞行座舱人因设计中的眼动追踪技术应用[J]. 包装工程, 2021, 42 (18): 84- 93.
	JI M, XIE X D, QIU Y. Application of eye tracking technology in flight cockpit human factors design[J]. Packaging Engineering, 2021, 42 (18): 84- 93.
27	DEY P, MADHVANATH S, RANJAN A, et al. An exploration of gesture-speech multimodal patterns for touch interfaces[C]// Proc. of the 3rd International Conference on Human Computer Interaction, 2011.
28	KIM M, SEONG E, JWA Y, et al. A cascaded multimodal natural user interface to reduce driver distraction[J]. IEEE Access, 2020, 8, 112969- 112984. doi: 10.1109/ACCESS.2020.3002775
29	SAKTHEESWARAN A, SRINIVASAN A, STASKO J. Touch? Speech? Or touch and speech? Investigating multimodal interaction for visual network exploration and analysis[J]. IEEE Trans. on Visualization and Computer Graphics, 2020, 26 (6): 2168- 2179. doi: 10.1109/TVCG.2020.2970512
30	VUKOVIC M, CAVEDON L, THANGARAJAH J, et al. Performance degrades less under increased workload with the addition of speech control in a dynamic environment[J]. Applied Ergonomics, 2021, 96, 103486. doi: 10.1016/j.apergo.2021.103486

窗口操作行为	任务场景	任务详情	按键交互实验	多模态交互实验
窗口操作行为	任务场景	任务详情	按键交互实验	触控手势	语音	触控手势+语音
切换窗口	态势	1.查看无人机是否到达任务区域1	单指点击	单指点击	—	—
	自卫	2.找到雷达告警系统	单指点击	—	“打开自卫”+“切换最大化/标准化”	—
	飞控	3.打开飞控查看无人机起飞，并切换为最大化	单指点击	—	—	“打开飞控”
移动窗口	火控	4.查看武器信息，并移到区域4	单指点击		—	—
	机电	5.移到区域3	单指点击	—	“打开机电”+“ 移到区域3”	—
	垂直	6.设置空速，并将其移到区域2	单指点击	—	—	“打开垂直”
关闭窗口	维护	7.打开维护全屏，再返回主菜单	单指点击		—	—
	协同	8.查看无人机2的载荷后关闭	单指点击	—	“无人机2信息”+ “关闭”	—
	协同	9.查看无人机3和无人机4的编队信息后关闭	单指点击	—	—	“回到主菜单”

多模态交互- 按键交互	个案数	秩平均值/s	秩的总和/s
负秩	$ 19^{a} $	12.58	239.00
正秩	$ 3^{b} $	4.67	14.00
绑定值	$ 0^{c} $	—	—

语音交互-按键交互	个案数	秩平均值/s	秩的总和/s
负秩	$ 12^{a} $	10.00	120.00
正秩	$ 10^{b} $	12.10	121.00
绑定值	$ 0^{c} $	—	—

触控手势语音结合交互-按键交互	个案数	秩平均值/s	秩的总和/s
负秩	$ 12^{a} $	11.36	136.20
正秩	$ 10^{b} $	10.60	106.00
绑定值	$ 0^{c} $	—	—

序号	问题
1	在起飞阶段查看高度、空速等信息时，我认为我会愿意使用该交互方式；
2	在使用机电系统时，我认为使用该交互方式操作各种功能很复杂；
3	在使用飞控控制无人机起飞时，我认为使用该交互方式容易操作；
4	查看态势信息时，我认为需要额外的帮助才能快速使用该交互方式；
5	在自卫阶段控制雷达告警系统时，我认为使用该交互方式容易感到混淆；
6	在查看武器信息时，我认为使用该交互方式进行操作很不流畅；
7	在查看无人机编队组网及载荷等信息时，我认为使用该交互方式感到自信。
8	在进行毁伤评估时，维护界面窗口比较多，我认为使用该交互方式操作很笨拙；
9	在整个有/无人机协同任务流程中，我认为大部分人会很快学会使用该交互方式完成任务；
10	在我可以使用该交互方式操作之前，我需要学习很多东西；

有/无人机协同操作界面的最佳交互方式评估

Assessing optimal interaction for a cooperative operation interface of manned/unmanned aerial vehicle

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 24

参考文献 30

相关文章 11

编辑推荐

Metrics

本文评价

维度	权重
脑力需求	0.267
体力需求	0.126
时间需求	0.229
努力程度	0.178
受挫程度	0.052
业绩水平	0.148

交互方式	按键交互	多模态交互
脑力需求	3.96	3.26
体力需求	2.18	2.35
时间需求	3.18	2.77
努力程度	4.54	3.54
受挫程度	2.61	2.07
业绩水平	3.21	2.96
总计	3.479	2.977

[1]	王纪凯, 豆亚杰, 李婧, 董奕君, 姜江, 谭跃进. 智能决策在军事体系工程的研究综述[J]. 系统工程与电子技术, 2025, 47(8): 2581-2599.
[2]	张睿文, 宋笔锋, 裴扬, 石帅. 基于ABMS的飞机拦截作战效能评估方法[J]. 系统工程与电子技术, 2018, 40(2): 322-329.
[3]	陈南宇, 黄俊, 周尧明, 赵文龙. 基于ESO的无人直升机轨迹鲁棒跟踪控制[J]. 系统工程与电子技术, 2018, 40(2): 368-374.
[4]	王超, 张胜修, 宋仔标, 杨建业, 吴晓露. 飞行器抗饱和鲁棒自适应非线性模型预测控制[J]. 系统工程与电子技术, 2018, 40(2): 393-400.
[5]	刘毅男,张胜修,曹立佳,张超. 基于LS-SVM的非线性鲁棒自适应飞行控制器设计[J]. Journal of Systems Engineering and Electronics, 2012, 34(12): 2541-2547.
[6]	程建锋,董新民,陈爱华,董志. 基于T-S模型的约束鲁棒保性能飞行控制律设计[J]. Journal of Systems Engineering and Electronics, 2012, 34(12): 2548-2553.
[7]	蔡红明, 昂海松, 邓双厚. 微型涵道飞行器的自适应逆控制方法[J]. Journal of Systems Engineering and Electronics, 2012, 34(3): 568-571.
[8]	高伟, 张鑫, 于飞, 冯晋. 基于观测量扩充的捷联惯导快速初始对准方法[J]. Journal of Systems Engineering and Electronics, 2011, 33(11): 2492-2495.
[9]	谢强，许录平，张华，孙景荣. 脉冲星导航解周期模糊匹配搜索算法[J]. Journal of Systems Engineering and Electronics, 2011, 33(11): 2496-2500.
[10]	张敏虎, 任章, 华春红. 惯性信息辅助的高动态弱GPS信号快速捕获[J]. Journal of Systems Engineering and Electronics, 2011, 33(2): 366-369.
[11]	杨志峰, 雷虎民, 李庆良, 李炯, 宋龙. 基于鲁棒自适应控制理论的导弹纵向逆控制[J]. Journal of Systems Engineering and Electronics, 2011, 33(2): 362-365.