Computer Science ›› 2017, Vol. 44 ›› Issue (Z6): 1-6.doi: 10.11896/j.issn.1002-137X.2017.6A.001

    Next Articles

Survey of Multimodal Delay Tele-interaction

WANG Hai-peng, HUANG Tian-biao, REN Chong-shuai and YAO Wu-yi   

  • Online:2017-12-01 Published:2018-12-01

Abstract: Multimodal Tele-interaction aims to use a variety of interactive modal and collaborative way and complementary characteristics and information between multiple modal,to communicate and understand the user interaction desires,improve the efficiency of interaction,enhance the naturalness of interaction,and enable users to complete the remote tasks in their “expectations”.Recently,with the applications of multimodal tele-interaction on the space exploration,deep sea exploitation and tele-surgery,the delay is introduced into the process of tele-interaction,which is produced mainly by the communication delay and bandwidth.The significant dalay causes the problems of asynchronous interaction and lack of interaction modal,which results a fundamental effect on the user’s behavior,psychological and cognitive characteristics,breaks and blocks the continuity,real-time and naturalness of interaction,not only degrade interactive user experience,but also is difficult to guarantee validity.The article defined the concept of multimodal tele-interaction,presented its key applications,and discussed the key technology,including the delay issue,asynchronous interaction and lack of interaction modal.Finally,we presented research challenges in the future.

Key words: Tele-interaction,Multimodal tele-interaction,Communication delay,Human factor,Asynchronous interaction

[1] 董士海,王坚,等.人机交互和多通道用户界面[M].北京:科学出版社,1999.
[2] 周芝庭,帅立国,等.互联网应用中触觉通信时延问题分析[J].测控技术,2008,27(4):53-55.
[3] VITTORIAS I,KAMMERL J,HIRCHE S,et al.Perceptual co-ding of haptic data in time-delayed teleoperation[C]∥EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems.2009:208-213.
[4] HIRCHE S,BUSS M.Transparent data reduction in networked telepresence and teleaction systems.part ii:Time-delayed communication[J].Presence:Teleoperators and Virtual Environments,2007,16(5):532-542.
[5] HINTERSEER P,STEIBACH E,CHAUDHURI S.Model baseddata compression for 3D virtual haptic teleinteraction[C]∥Digest of Technical Papers.International Conference on Consumer Electronics.2006:23-24.
[6] SUOMELA J.Tele-presence Aided Teleoperation of Semi-au-tonomous Work Vehicles[D].Helsinki University of Technology,2001.
[7] 朱士俊.远程医学的建立和发展[J].解放军医院管理,1998,5(3):297.
[8] 张珩,李庚田.遥科学的概念、 应用与发展[J].中国航天,1997(11):15-20.
[9] SATO K,HISATAKA T.Unmanned system for heavy con-struction equipment[C]∥Proc.of the 5th Int.Symp.on Robo-tics in Construction.Tokyo,Japan,1988:449-458.
[10] FERRELL W R.Remote manipulation with transmission delay[J].IEEE Transactions on Human Factors in Electronics,1965 (1):24-32.
[11] PAVLOVYCH,ANDRIY,STUERZLINGER W.Target following formance in the presence of latency,jitter,and signal dropouts[C]∥Proceedings of Graphics Interface Canadian Human-Computer Communications Society.2011:33-40.
[12] PANTEL L,WOLF L C.On the impact of delay on real-timemultiplayer games[C]∥Proceedings of the 12th International Workshop on Network and Operating Systems Support for Di-gital Audio and Video.ACM,2002:23-29.
[13] KAHNEMAN D.Attention and effort[M].Englewood Cliffs,NJ:Prentice-Hall,1973.
[14] 黄攀峰,刘正雄.空间遥操作技术[M].长沙:国防工业出版社,2015.
[15] YORIZAWA I,TAKIKAWA K,OKUBO S,et al.Subjective ef-fects of talker echo and transmission delay in video conferencing via communication satellite[J].Trans.IECE Japan,1981,64(11):1281-1288.
[16] LIU Z,HEER J.The effects of interactive latency on exploratory visual analysis[J].IEEE Transactions on Visualization and Computer Graphics,2014,20(12):2122-2131.
[17] KIM T,ZIMMERMAN P M,WADE M J,et al.The effect of delayed visual feedback on telerobotic surgery[J].Surgical Endoscopy And Other Interventional Techniques,2005,19(5):683-686.
[18] Smith K U,WARGO L,JONES R,et al.Delayed and space-displaced sensory feedback and learning[J].Perceptual and Motor Skills,1963,16(3):781-796.
[19] FERRELL W R,SHERIDAN T B.Remote manipulative control with transmission delay[J].IEEE Transactions on Human Factors in Electronics,1963,HFE-4(1):25-29.
[20] SMITH W M,MCCRAY J W,SMITH K U.Delayed visual feedback and behavior[J].Science,1960,132(3433):1013-1014.
[21] BRUTLAG J.Speed matters for Google web search[R].Google.June,2009.
[22] JOTA R,NG A,DIETZ P,et al.How fast is fast enough?:a study of the effects of latency in direct-touch pointing tasks[C]∥Proceedings of the SIGCHI Conference on Human Factors in Computing Systems.ACM,2013:2291-2300.
[23] MACKENZIE IS,WARE C.Lag as a determinant of human performance in interactive systems[C]∥Proceedings of the INTERACT’93 and CHI’93 Conference on Human Factors in Computing Systems.ACM,1993:488-493.
[24] MIYASATO T,NAKATSU R.Allowable delay between images and tactile information in a haptic interface[C]∥International Conference on Virtual Systems and MultiMedia,1997.IEEE,1997:84-89.
[25] JAY C,GLENCROSS M,HUBBOLD R.Modeling the effects of delayed haptic and visual feedback in a collaborative virtual environment[J].ACM Transactions on Computer-Human Interaction (TOCHI),2007,14(2):8.
[26] KAARESOJA T,ANTTILA E,HOGGAN E.The effect oftactile feedback latency in touchscreen interaction[C]∥2011 IEEE World Haptics Conference (WHC).IEEE,2011:65-70.
[27] HIRZINGER G,BRUNNER B,DIETRICH J,et al.ROTEX-the first remotely controlled robot in space[C]∥1994 IEEE International Conference on Robotics and Automation,1994.IEEE,1994:2604-2611.
[28] ANVARI M,BRODERICK T,STEIN H,et al.The impact of latency on surgical precision and task completion during robotic-assisted remote telepresence surgery[J].Computer Aided Surgery,2005,10(2):93-99.
[29] LEWIS C H,GRIFFIN M J.Human factors consideration inclinical applications of virtual reality[J].Studies in Health Technology and Informatics,1997:35-58.
[30] LAWTHER A,GRIFFIN M J.The motion of a ship at sea and the consequent motion sickness amongst passengers[J].Ergonomics,1986,29(4):535-552.
[31] HETTINGER L J,RICCIO GE.Visually induced motion sickness in virtual environments[J].Presence:Teleoperators & Virtual Environments,1992,1(3):306-310.
[32] WLOKA M M.Lag in multiprocessor virtual reality[J].Pre-sence:Teleoperators & Virtual Environments,1995,4(1):50-63.
[33] LIU A,THARP G,FRENCH L,et al.Some of what one needs to know about using head-mounted displays to improve tele-operator performance[J].IEEE Transactions on Robotics and Automation,1993,9(5):638-648.
[34] THARP G K,LIU AM,FRENCH L,et al.Timing considerations of helmet-mounted display performance[C]∥SPIE/IS&T 1992 Symposium on Electronic Imaging:Science and Technology.International Society for Optics and Photonics,1992:570-576.
[35] LIU A,THARP G,HIROSE M,et al.Visual factors affectinghuman operator performance with a helmet-mounted display[R].SAE Technical Paper,1991.
[36] HELD R,DURLACH N.Telepresence,time delay and adapta-tion[J].Pictorial Communication in Virtual and Real Environments,1991:232-246.
[37] DWYER T F.Telepsychiatry:psychiatric consultation by interactive television[J].American Journal of Psychiatry,1973,130(8):865-869.
[38] NORDAHL R.Self-induced footsteps sounds in virtual reality:Latency,recognition,quality and presence[C]∥The 8th Annual International Workshop on Presence(PRESENCE 2005).2005:353-355.
[39] MOUTOUSSIS K.Functional segregation and temporal hierarchy of the visual perceptive systems[J].Proceedings of the RoyalSociety of London B:Biological Sciences,1997,264(1387):1407-1414.
[40] FERRELL W R.Remote manipulation with transmission delay[J].IEEE Transactions on Human Factors in Electronics,1965,(1):24-32.
[41] DIXON N F,SPITZ L.The detection of auditory visual desynchrony[J].Perception,1980,9(6):719-721.
[42] SHI Z,ZOU H,MULLER H J.Temporal perception of visual-haptic events in multimodal telepresence system[M].In Tech,2010.
[43] VATAKIS A,SPENCE C.Evaluating the influence of framerate on the temporal aspects of audiovisual speech perception[J].Neuroscience Letters,2006,405(1):132-136.
[44] VAN ERP J B F,WERKHOVEN P J .Vibro-tactile and visual asynchronies:Sensitivity and consistency[J].Perception,2004,33(1):103-111.
[45] FJISKAI W,SHIMOJO S,KASHINO M,et al.Recalibration of audiovisual simultaneity[J].Nature Neuroscience,2004,7(7):773-778.
[46] STONE J V,HUNKIN N M,PORRILL J,et al.When is now? Perception of simultaneity[J].Proceedings of the Royal Society of London B:Biological Sciences,2001,268(1462):31-38.
[47] SPENCE C,SHORE D I,KLEIN R M.Multisensory prior entry[J].Journal of Experimental Psychology:General,2001,130(4):799.
[48] DIXON N F,SPITZ L.The detection of auditory visual desynchrony[J].Perception,1980,9(6):719-721.
[49] LEVITIN D J,MACLEAN K,MATHEWS M,et al.The perception of cross-modal simultaneity[J].Proc.of International Journal of Computing Auticipatory Systems,2008,517(1):323-329.
[50] VOGELS I M L C.Detection of temporal delays in visual-haptic interfaces[J].Human Factors:The Journal of the Human Factors and Ergonomics Society,2004,46(1):118-134.
[51] THOMPSON JM,OTTENSMEYER M P.Human factors in telesurgery:effects of time delay and asynchrony in video and control feedback with local manipulative assistance[J].Telemed J,1999,5(2):29-37.
[52] SPENCE C,SQUIRE S.Multisensory integration:maintaining the perception of synchrony[J].Curr Biol,2003,13(13):R519-R521.
[53] ANDERSON R J,SPONG MW.Bilateral control of teleoperators with time delay[J].IEEE Transactions on Automatic Control,1989,34(5):494-501.
[54] KIM T,ZIMMERMAN P M,WADE M J,et al.The effect of delayed visual feedback on telerobotic surgery[J].Surgical Endoscopy And Other Interventional Techniques,2005,19(5):683-686.
[55] ZAMPINI M,SHORE D I,SPENCE C.Audiovisual temporalorder judgments[J].Experimental Brain Research,2003,152(2):198-210.
[56] SPENCE C,SHORE D I,KLEIN R M.Multisensory prior entry[J].Journal of Experimental Psychology:General,2001,130(4):799-832.
[57] ZAMPINI M,BROWN T.Audiotactile temporal order judg-ments[J].Acta Psychologica,2005,118(3):277-291.
[58] BRANDI F,KAMMERL J,STEINBACH E.Error-resilient perceptual coding for networked haptic interaction[C]∥Procee-dings of the 18th ACM International Conference on Multimedia.ACM,2010:351-360.
[59] SHI Z,ZOU H,RANK M,et al.Effects of packet loss and latency on the temporal discrimination of visual-haptic events[J].IEEE Transactions on Haptics,2010,3(1):28-36.
[60] FJISAKI W,SHIMOJO S,KASHNO M,et al.Recalibration of audiovisual simultaneity[J].Nature Neuroscience,2004,7(7):773-778.
[61] TAM J,CARTER E,KIESLER S,et al.Video increases the perception of naturalness during remote interactions with latency[C]∥Extended Abstracts on Human Factors in Computing Systems(CHI’12).ACM,2012:2045-2050.
[62] VITTORIALS I,KAMMERL J,HIRCHE S,et al.Perceptualcoding of haptic data in time-delayed teleoperation[C]∥EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems(World Haptics 2009).Third Joint,IEEE,2009:208-213.
[63] HIRCHE S,HINTERSEER P,STEINBACH E,et al.Transpa-rent data reduction in networked telepresence and teleaction systems.part i:Communication without time delay[J].Presence:Teleoperators and Virtual Environments,2007,16(5):523-531.
[64] 俸文.多通道人机交互技术的研究[D].南京:南京理工大学,2004.
[65] TAKAHASHI Y,MASUDA I.A visual interface for securityrobots[C]∥IEEE International Workshop on Robot and Human Communication,1992.IEEE,1992:123-128.
[66] LIN Q,KUO C.Virtual tele-operation of underwater robots[C]∥1997 IEEE International Conference on Robotics and Automation,1997.IEEE,1997,2:1022-1027.
[67] 徐旭明,叶榛.基于视觉临场感的机器人遥操作系统[J].高技术通讯,2000(3):57-61.
[68] 艾海舟,张朋飞.室外移动机器人的视觉临场感系统[J].机器人,2000,22(1):28-32.
[69] IMAIDA T,YOKOKOHJI Y,DOI T,et al.Ground-space bila-teral teleoperation of ETS-VII robot arm by direct bilateral coupling under 7-s time delay condition[J].IEEE Transactions on Robotics and Automation,2004,20(3):499-511.
[70] HELD R,EFSTATHIOU A,GREENE M,et al.Adaptation to Displaced and Delayed Visual Feedback from the Hand[J].Journal of Experimental Psychology:General,1966,2(6):887-891.
[71] BUSS M,PEER A,SCHAUSS T,et al.Multi-modal multi-user telepresence and teleaction system[C]∥IEEE/RSJ International Conference on Intelligent Robots and Systems,2008(IROS 2008).IEEE,2008:4137-4138.
[72] USS M,PEER A,SCHAUSS T,et al.Development of a Multi-modal Multi-user Telepresence and Teleaction System[J].The International Journal of Robotics Research,2010,29(10):1298-1316.
[73] FERRARI E,CAVALLO E.Operator-machine interaction:ergonomic evaluation of a vineyard teleoperated machine[C]∥International Conference of Agricultural Engineering-CIGR-AgEng 2012:Agriculture and Engineering for a Healthier Life.Valencia,Spain,CIGR-EurAgEng,2012.
[74] STASSEN H G,SMETS G J F.Telemanipulation and Telepre-sence[J].Control Eng.Practice,1997,5(3):363-374.
[75] 徐旭明,叶棒.基于视觉临场感的机器人遥操作系统[J].高技术通讯,2000,10(3):57-60.
[76] 戴炬.异机构遥控主从手的双向力反映[J].控制理论和应用,1996(13):259-261.
[77] 游松.基于国际互联网的遥操作机器人系统的研究[D].北京:北京航空航天大学,2000.
[78] 吴国钊,傅雪冬.基于Intemet的机器人实时跟踪系统[J].机器人技术与应用,2000(1):24-26.
[79] 马宏鹏,安刚.基于互联网的机器人遥操作系统平台[J].自动化与仪表,2000,15(6):3-7.
[80] 韦庆,李杰.基于事件反馈的机器人监控技术及其在大时延遥操作中的应用[C]∥空间机器人及遥科学技术研讨会论文集.1999:307-313.
[81] 陈辉,宋爱国.带有力觉和触觉临场感的灵巧手主从系统的设计[J].机器人,1998,20(6):437-441.
[82] 宋爱国,曾庆军.FTP技术中多传感器信号的无源传输算法[J].电子学报,1996,24(11):108-112.
[83] 刘准.国外深海技术发展研究[J].船艇,2006(10):6-8.
[84] SAGER W W,SANO T,GELDMACHER J.IODP Expedition324:Ocean Drilling at Shatsky Rise Gives Clues about Oceanic Plateau Formation[J].Scientific Drilling,2011(12):24-31.
[85] GUPTA H,NAYAK S,COMMITTEE K W.Deep scientificdrilling to study reservoir-triggered Earthquakes in Koyna,Western India[J].Scientific Drilling,2011,12:53-54.
[86] 莫杰,肖菲.深海探测技术的发展[J].科学,2012,4(5):11-15.
[87] 徐文,钱洪宝,张杰,等.我国海洋监测高技术发展的回顾与思考[J].海洋技术学报,2015,34(3):59-61.
[88] LABONTE D,BOISSY P,MICHAUD F.Comparative analysis of 3-D robot teleoperation interfaces with novice users[J].IEEE Transactions on Systems,Man,and Cybernetics(Part B),2010,40(5):1331-1342.
[89] HUBER D,HERMAN H,KELLY A,et al.Real-time photo-rea-listic visualization of 3D environments for enhanced tele-operation of vehicles[C]∥IEEE 12th International Conference on Computer Vision Workshops (ICCV Workshops),2009.IEEE,2009:1518-1525.
[90] KIM S,KANG J,CHUNG M J.Probabilistic voxel mappingusing an adaptive confidence measure of stereo matching[J].Intelligent Service Robotics,2013,6(2):89-99.
[91] 马厦飞,连琏.“海马”号 4500 米级 ROV 系统研发历程[J].船舶与海洋工程,2015,31(1):9-12.
[92] 高艳波,李慧青.深海高技术发展现状及趋势[J].海洋技术,2010,29(3):119-124.
[93] DWYER T F.Telepsychiatry:psychiatric consultation by interactive television[J].American Journal of Psychiatry,1973,130(8):865-869.
[94] SINGH R,JEONG B,RENAMBOTET L.TeraVision:a distributed,scalable,high resolution graphics streaming system[C]∥2004 IEEE International Conference on Cluster Computing.IEEE,2004:391-400.
[95] JEONG B.Collaborative Visualization Architeeturein Scalable A-daptive Graphics Environment[C]∥IBM Visualization and Graphics Student Symposium,TJ Watson Research Hawthorne and Yorktown.2007.
[96] FRIESEN J A,TARMAN T D.Remote high-performance visua-lization and collaboration[J].IEEE Computer Graphics and Applications,2000,20(4):45-49.
[97] YHAKUR ,BAX J S,HOLDSWORTH D W,et al.Design and performance evaluation of a remote catheter navigation system[J].IEEE Transactions on Biomedical Engineering,2009,56(7):1901-1908.
[98] 刘浩.导管机器人系统的建立及其关键技术研究[D].哈尔滨:哈尔滨工业大学,2010.
[99] 李显凌.用于微创介入手术的导管导向机器人研究[D].哈尔滨:哈尔滨工业大学,2009.
[100] RAHMAN S M M,IKEURA R,NOBE M,et al.Control of a power assist robot for lifting objects based on human operator’s perception of object weight[C]∥The 18th IEEE International Symposium on Robot and Human Interactive Communication,2009(RO-MAN 2009).IEEE,2009:84-90.
[101] WATANABE T,YOSHIKAWA T.Grasping optimization using a required external force set[J].IEEE Transactions on Automation Science and Engineering,2007,4(1):52-66.
[102] SHELTON J N,CHIU G T C,PIZLO Z.Exponentially segmented positioning of a single link mechanism:A control algorithm that satisfies Fitts’ Law[C]∥American Control Conference,2007(ACC’07).IEEE,2007:5983-5988.
[103] 熊友军.基于增强现实的遥操作关键技术研究[D].武汉:华中科技大学,2005.

No related articles found!
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!