计算机科学 ›› 2022, Vol. 49 ›› Issue (6): 66-72.doi: 10.11896/jsjkx.220400034

• 6G 赋能智慧物联网技术与应用* 上一篇    下一篇

面向6G的全景视频片划分优化编码算法

杨桃雨, 徐媛媛, 谭增洁   

  1. 河海大学计算机与信息学院 南京 211100
  • 收稿日期:2022-04-02 修回日期:2022-04-24 出版日期:2022-06-15 发布日期:2022-06-08
  • 通讯作者: 徐媛媛(yuanyuan_xu@hhu.edu.cn)
  • 作者简介:(yty@hhu.edu.cn)
  • 基金资助:
    国家自然科学基金(61801125)

Tile Partition Optimized Omnidirectional Video Coding for 6G Network

YANG Tao-yu, XU Yuan-yuan, TAN Zeng-jie   

  1. School of Computer and Information,Hohai University,Nanjing 211100,China
  • Received:2022-04-02 Revised:2022-04-24 Online:2022-06-15 Published:2022-06-08
  • About author:YANG Tao-yu,born in 1998,master,is a member of China Computer Federation.Her main research interests include immersive videos and video coding.
    XU Yuan-yuan,born in 1983,Ph.D,associate professor,postgraduate supervisor,is a member of China Computer Federation.Her main research interests include image/video coding,image/video communication,immersive videos,visual perception and applications.
  • Supported by:
    National Natural Science Foundation of China (61801125).

摘要: 第6代无线通信的兴起为虚拟现实全景视频的发展提供了更广阔的前景。基于片的全景视频编码方案能够在相同网络带宽条件下提升全景视频的观看体验,片划分大小会影响视频传输的性能,与较小的片相比,采用较大的片能够降低编码冗余,但需传送更大区域来覆盖相同视口,这会造成更多的像素开销。目前已有片划分工作主要是针对矩形视口而设计的,而实际中,球型视频到二维平面的投影会对视口不同区域进行不同程度的拉伸,用矩形片覆盖不规则视口区域时像素开销计算更为复杂。针对这一挑战,文中提出了一种适用于用户真实视口的片划分算法。首先,分析了投影格式对视口区域造成的拉伸失真,并对在不同片划分下不规则视口区域的额外像素开销进行了推导;然后,通过权衡不同划片粒度编码单元的像素开销与编码效率的关系,提出了一种全景视频序列最优片划分编码方案;最后,通过与片划分穷举搜索法进行对比,验证了所提算法在较少的计算复杂度下能取得与穷举搜索法相当的传输效率。

关键词: 6G, 感知互联, 片划分, 全景视频, 视频编码

Abstract: The rise of the 6G wireless communication provides a broader prospect for the development of virtual reality panoramic video.The tile-based panoramic video coding scheme can improve the viewing experience of 360° video under the same condition of network bandwidth.Tile partition affects the video transmission performance.Compared with small tiles,using large tiles can effectively improve coding efficiency,but it will cause more pixel overhead by transmitting a larger area to cover the viewport.The existing tile partition work is mainly designed for the rectangular viewport,but the projection of spherical video to a two-dimensional plane will stretch different areas of the viewport to varying degrees in practice.Deriving the pixel overhead associated with covering irregular viewport areas with rectangular tiles is more complicated.To address this challenge,a tile partitioning algorithm for the user's real viewport has been proposed in this paper.Firstly,the stretching distortion of the viewport caused by the projection format is analyzed,and the pixel overhead of the irregular viewport with different tile partition sizes is derived.Secondly,by trading off the pixel overhead and the coding efficiency of tiles with different granularity,an optimal tile partition scheme has been proposed for the panoramic video sequence.Finally,the proposed scheme is compared with the exhaustive search method for tile partition in the experiment,and the results show that the proposed algorithm can achieve almost the same transmission efficiency as the exhaustive search method with less computational complexity.

Key words: 6G networks, Omnidirectional video, Sensory interconnection, Tile partition, Video coding

中图分类号: 

  • TP751.1
[1] AHMAD A,MANSOOR A,BARAKABITZE A,et al.Supervised-learning-Based QoE Prediction of Video Streaming in Future Networks:A Tutorial with Comparative Study[J].IEEE Communications Magazine,2021,59(11):88-94.
[2] ALWIS C D,ANSHUMAN K,PHAM Q V,et al.Survey on 6G Frontiers:Trends,Applications,Requirements,Technologies and Future Research[J].IEEE Open Journal of the Communications Society,2021,2:836-886.
[3] NGUYEN D C,MING D,PATHIRANA P N,et al.6G Internet of Things:A Comprehensive Survey[J].IEEE Internet of Things Journal,2022,9(1):359-383.
[4] KAUL A,GUPTA J.Revolutionary 6G:Technologies,Archi-tecture,Coverage,and Performance[C]//Proceedings of International Conference on Computing Communication and Net-working Technologies.New York:IEEE Press,2021:1-6.
[5] SKUPIN R,SANCHEZ Y,HELLGE C,et al.Tile Based HEVC Video for Head Mounted Displays[C]//Proceedings of IEEE International Symposium on Multimedia.California:IEEE Press,2016:399-400.
[6] CHIARIOTTI F.A Survey on 360-Degree Video:Coding,Qua-lity of Experience and Streaming[J].Computer Communications,2021,177(1):133-155.
[7] SREEDHAR K K,AMINLOU A,HANNUKSELA M M,et al.Viewport-Adaptive Encoding and Streaming of 360-Degree Video for Virtual Reality Applications[C]//IEEE International Symposium on Multimedia.New York:IEEE Press,2016:583-586.
[8] NASRABADI A T,MAHZARI A,BESHAY J D,et al.Adaptive 360-Degree Video Streaming Using Layered Video Coding[C]//IEEE Virtual Reality.California:IEEE Press,2017:347-348.
[9] VISHWANATH B,NANJUNDASWAM T,ROSE K.Rota-tional Motion Model for Temporal Prediction in 360 video coding[C]//IEEE 19th International Workshop on Multimedia Signal Processing.United Kingdom:IEEE Press,2017:1-6.
[10] FREMEREY S,SINGLA A,MESEBERG K.Avtrack360:An Open Dataset and Software Recording People’s Head Rotations Watching 360° Videos on an HMD[C]//ACM Multimedia System Conference.New York:ACM Press,2018:403-408.
[11] XU M,LI C,ZHANG S,et al.State-of-the-Art in 360° Video/Image Processing:Perception,Assessment and Compression[J].IEEE Journal of Selected Topics in Signal Processing,2020,14(1):5-26.
[12] SULLIVAN G J,OHM J R,HAN W J,et al.Overview of The High Efficiency Video Coding (HEVC) Standard[J].IEEE Transactions on circuits and systems for video technology,2012,22(12):1649-1668.
[13] LI L,YAN N,LI Z,et al.λ-Domain Perceptual Rate Control for 360-Degree Video Compression[J].IEEE Journal of Selected Topics in Signal Processing,2020,14(1):130-145.
[14] ZHOU Y,TIAN L,ZHU C,et al.Video Coding Optimization for Virtual Reality 360-Degree Source[J].IEEE Journal of Selected Topics in Signal Processing,2020,14(1):118-129.
[15] XU Y,YANG T,TAN Z,et al.FoV-based Coding Optimization for 360-degree Virtual Reality Videos[C] //Proceedings of IEEE International Conference on Acoustics,Speech and Signal Processing.Singapore:IEEE Press,2022.
[16] LI Y,XU J,CHEN Z.Spherical Domain Rate-Distortion Optimization for 360-Degree Video Coding[C] //Proceedings of IEEE International Conference on Multimedia and Expo.Hong Kong:IEEE Press,2017:709-714.
[17] LIU Y,GUO H,ZHU C,et al.Spherical Position DependentRate-Distortion Optimization for 360-degree Video Coding[C]//Proceedings of Asia-Pacific Signal and Information Processing Association Annual Summit and Conference.Lanzhou:IEEE Press,2019:992-996.
[18] CHIANG J,YANG C,DEDHIA B,et al.Saliency-Driven Rate-Distortion Optimization for 360-Degree Image Coding[J].Multimedia Tools and Applications,2021,80(6):8309-8329.
[19] YANG C,CHIANG J,LIE W.Rate-Distortion Optimization for 360-degree Image Considering Visual Attention[C]//Asia-Pacific Signal and Information Processing Association Annual Summit and Conference.Los Alamitos:IEEE Computer Society,2020:1128-1131.
[20] SUN Y,YU L.Coding Optimization Based on Weighted-to-Spherically-Uniform Quality Metric for 360 Video[C]//Proceedings of IEEE Visual Communications and Image Processing.USA:IEEE Press,2017:1-4.
[21] LUZ G,ASCENSO J,BRITES C,et al.Saliency-Driven Omnidirectional Imaging Adaptive Coding:Modeling and Assessment[C]//Proceedings of IEEE 19th International Workshop on Multimedia Signal Processing.United Kingdom:IEEE Press,2017:1-6.
[22] XIAO M,ZHOU C,LIU Y,et al.Optile:Toward Optimal Tiling in 360-Degree Video Streaming[C]//Proceedings of the 25th ACM International Conference on Multimedia.New York:ACM Press,2017:708-716.
[23] YU M,LAKSHMAN H,GIROD B.Content Adaptive Repre-sentations of Omnidirectional Videos for Cinematic Virtual Reality[C]//Proceedings of the 3rd International Workshop on Immersive Media Experiences.New York:ACM Press,2015:1-6.
[24] MAVLANKAR A,GIROD B.Spatial-Random-Access-Enabled Video Coding for Interactive Virtual Pan/Tilt/Zoom Functionality[J].IEEE Transactions on Circuits and Systems for Video Technology,2011,21(5):577-588.
[25] GUAN Y,ZHENG C,ZHANG X,et al.Pano:Optimizing 360 Video Streaming with A Betterunderstanding of Quality Perception[C]//Proceedings of the ACM Special Interest Group on Data Communication.New York:ACM Press,2019:394-407.
[26] SANCHEZ Y,SKUPIN R,HELLGE C,et al.Spatio-Temporal Activity based Tiling for Panorama Streaming[C]//Proceedings of the 27th Workshop on Network and Operating Systems Support for Digital Audio and Video.New York:ACM Press,2017:61-66.
[27] OZCINAR C,CABRERA J,SMOLIC A.Omnidirectional Video Streaming Using Visual Attention-Driven Dynamic Tiling for VR[C]//IEEE Visual Communications and Image Processing.New York:IEEE Press,2018:1-4.
[28] KAN N,ZOU J,LI C,et al.RAPT360:Reinforcement Learning-Based Rate Adaptation For 360-Degree Video Streaming with Adaptive Prediction and Tiling[J].IEEE Transactions on Circuits and Systems for Video Technology,2021,67(2):409-423.
[29] GUO H W,LUO H J,LIU S,et al.Improved R-λ Model Based Rate Control Algorithm[J].Computer Science,2019,46(3):142-147.
[30] CHEN L,GAO L,REN J,et al.Adaptive Bitrate Streaming for Energy-Efficiency Mobile Augmented Reality[J].Computer Science,2022,49(1):194-203.
[31] OZCINAR C,CABRERA J,SMOLIC A.Visual Attention-Aware Omnidirectional Video Streaming Using Optimal Tiles for Virtual Reality[J].IEEE Journal on Emerging and Selected Topics in Circuits and Systems,2019,9(1):217-230.
[32] NGUYEN D V,TRAN H,THANG T.Adaptive Tiling Selection for Viewport Adaptive Streaming of 360-degree Video[J].IEICE Transactions on Information and Systems,2019,102(1):48-51.
[33] KATTADIGE C,THILAKARATHNA K.VAD360:Viewport Aware Dynamic 360-Degree Video Frame Tiling[J].arXiv:2105.11563,2021.
[34] JEAN L,CONCOLATO C.Tiled-Based Adaptive StreamingUsing MPEG-DASH[J].Multimedia Systems,2016,41(3):1-3.
[1] 曲倩文, 车啸平, 曲晨鑫, 李瑾如.
基于信息感知的虚拟现实用户临场感研究
Study on Information Perception Based User Presence in Virtual Reality
计算机科学, 2022, 49(9): 146-154. https://doi.org/10.11896/jsjkx.220500200
[2] Ran WANG, Jiang-tian NIE, Yang ZHANG, Kun ZHU.
Clustering-based Demand Response for Intelligent Energy Management in 6G-enabled Smart Grids
Clustering-based Demand Response for Intelligent Energy Management in 6G-enabled Smart Grids
计算机科学, 2022, 49(6): 44-54. https://doi.org/10.11896/jsjkx.220400002
[3] 金华, 朱靖宇, 王昌达.
视频隐私保护技术综述
Review on Video Privacy Protection
计算机科学, 2022, 49(1): 306-313. https://doi.org/10.11896/jsjkx.201200047
[4] 成昭炜, 沈航, 汪悦, 王敏, 白光伟.
基于深度强化学习的无人机辅助弹性视频多播机制
Deep Reinforcement Learning Based UAV Assisted SVC Video Multicast
计算机科学, 2021, 48(9): 271-277. https://doi.org/10.11896/jsjkx.201000078
[5] 吉晓祥, 沈航, 白光伟.
异构无线网络中基于非正交多址的可伸缩视频多播机制
Non-orthogonal Multiple Access Enabled Scalable Video Multicast in HetNets
计算机科学, 2021, 48(11): 356-362. https://doi.org/10.11896/jsjkx.200900080
[6] 蔡于涵,熊淑华,孙伟恒,Karn Pradeep,何小海.
基于运动矢量细化的帧率上变换与HEVC结合的视频压缩算法
Video Compression Algorithm Combining Frame Rate Up-conversion with HEVC Standard Based on Motion Vector Refinement
计算机科学, 2020, 47(2): 76-82. https://doi.org/10.11896/jsjkx.190500092
[7] 何晓艺,段凌宇,林巍峣.
基于深度残差网络的HEVC压缩视频增强
Deep Residual Network Based HEVC Compressed Videos Enhancement
计算机科学, 2019, 46(3): 88-91. https://doi.org/10.11896/j.issn.1002-137X.2019.03.011
[8] 郭红伟, 骆洪军, 刘帅, 牛林, 杨波.
一种改进的R-λ模型码率控制算法
Improved R-λ Model Based Rate Control Algorithm
计算机科学, 2019, 46(3): 142-147. https://doi.org/10.11896/j.issn.1002-137X.2019.03.021
[9] 杜秀丽, 胡兴, 陈波, 邱少明.
基于加权非局部相似性的视频压缩感知多假设重构算法
Multi-hypothesis Reconstruction Algorithm of DCVS Based on Weighted Non-local Similarity
计算机科学, 2019, 46(1): 291-296. https://doi.org/10.11896/j.issn.1002-137X.2019.01.045
[10] 任云,程福林,黎洪松.
基于频率敏感三维自组织映射的视差估计算法
Disparity Estimation Algorithm Based on Frequency Sensitive Three-dimensional Self-organizing Map
计算机科学, 2017, 44(Z11): 225-227. https://doi.org/10.11896/j.issn.1002-137X.2017.11A.047
[11] 单娜娜,周巍,段哲民.
HEVC中的变换系数熵编码优化算法
Improved Entropy Coding Algorithm for Transform Coefficients in HEVC
计算机科学, 2017, 44(6): 290-293. https://doi.org/10.11896/j.issn.1002-137X.2017.06.051
[12] 张培君,金小娟,王淑慧,周开伦,林涛.
结合全色度HEVC和有损字典算法的屏幕图像编码
Screen Content Coding of Combining Full-chroma HEVC and Lossy Matching Dictionary Coder
计算机科学, 2014, 41(3): 286-292.
[13] 陈卓,冯钢,陆毅.
一种分层P2 P流媒体系统重叠网的构建策略
Overlay Construction Policy in Layered P2P Streaming System
计算机科学, 2012, 39(5): 69-74.
[14] 张晓星,刘冀伟,胡广大,崔朝辉.
分布式视频编码中的边信息改进算法
Improved Side Information Generation Algorithm in Distributed Video Coding
计算机科学, 2011, 38(11): 275-277.
[15] 陆寄远,孙少晖,朝红阳.
联合分数运动估计优化的多参考帧快速选择算法
Fast Algorithm for Multi-frame Selection with Jointly Optimization of Fractional Pixel Motion Estimation
计算机科学, 2010, 37(6): 283-285.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!