基于SIFT算法的大场景视频拼接算法及优化

doi:10.11896/j.issn.1002-137X.2019.07.044

摘要/Abstract

摘要： 目前大量的由独立视频设备获取的小场景视频信息难以满足大场景下信息处理的要求,而通过多设备人工查阅的方式又存在效率低下、信息冗余和碎片化等问题。文中研究了大场景视频拼接技术,利用SIFT算法的尺度不变特性对关键点进行特征匹配,通过仿射矩阵变形完成对图像的拼接工作。在此过程中,对传统的SIFT拼接算法进行进一步的优化,主要是基于距离的优化算法来完善视频拼接的效果;对SIFT特征点匹配、加权优化算法、关键帧提取的技术等进行并行加速,以提高拼接效率。实验结果表明,提出的优化方法能更好地提取视频中的关键信息,以实现更好的视频拼接效果。在视觉效果上,所提方法得到的拼接结果中不存在传统方法出现的两幅图像的交接线。此外,在MATLAB环境下分别对关键点检测和拼接部分进行了加速优化,优化后的关键点检测效率提高了约20%,拼接部分的效率提高了将近57%。在C++环境下,关键点的检测效率提升了14%,拼接部分的检测效率提升了40%。

关键词: SIFT特征点匹配, 并行优化, 关键帧, 加权优化算法, 视频拼接

Abstract: At present,a large number of video information obtained by independent video devices in small scenes cannot meet the requirements of information processing in large scenes,and the manual access by multiple devices is faced with such problems as low efficiency,information redundancy and fragmentation.This paper investigated large scene video stitching technology,carried out the feature matching of key points by using the scale-invariant feature of SIFT algorithm,and completed the image splicing by affine matrix deformation in this process.The traditional SIFT splicing algorithm is further optimized,mainly based on distance optimization algorithm to improve the effect of video splicing.In order to improve the efficiency of stitching,the techniques of key frame extraction based on SIFT feature point matching weighted optimization algorithm were accelerated in parallel.Experiments show that the proposed optimization method can extract essential information in the video more effectively,achieving better video stitching results.Visually,the intersection of the two images appeared in the results of the conventional method does not appear in the stitching results obtained by the proposed method.Moreover,the key point detection and splicing parts were accelerated and optimized respectively in the different environment.In MATLAB,the efficiency of the key point with optimized is improved by 20%,and that of splicing parts is increased by about 57%.In C++,the efficiency of the key point is improved by 14%,and that of splicing parts is increased by about 40%.

Key words: Key frame, Parallel optimization, SIFT feature point matching, Video Stitching, Weighted optimization algorithm

中图分类号:

TP391

杨思燕,贺国旗,刘如意. 基于SIFT算法的大场景视频拼接算法及优化[J]. 计算机科学, 2019, 46(7): 286-291. https://doi.org/10.11896/j.issn.1002-137X.2019.07.044

YANG Si-yan,HE Guo-qi,LIU Ru-yi. Video Stitching Algorithm Based on SIFT and Its Optimization[J]. Computer Science, 2019, 46(7): 286-291. https://doi.org/10.11896/j.issn.1002-137X.2019.07.044

参考文献

[1]ZHANG J,CHEN G,JIA Z.An image stitching algorithm based on histogram matching and SIFT algorithm [J].International Journal of Pattern Recognition and Artificial Intelligence,2017,31(4):1754006.
[2]ZHENG G L,TANG B B.The stereo vision positioning system based on multiple fisheye cameras [J].Computer Simulation,2016,33(7):256-260.(in Chinese)
郑贵林,唐贝贝.基于多鱼眼摄像头的立体视觉定位系统[J].计算机仿真,2016,33(7):256-260.
[3]KAUR S,ER S K.Analysis of Image Stitching for Noisy Images using SIFT [J].International Journal of Advanced Research in Computer Science,2017,8(5):2078-2082.
[4]WANG Y W,YU M,JIANG H,et al.An image stitching algorithm via adaptive quadtree segmentation[J].Journal of Ningbo University (Natural Science & Engineering),2018,31(4):52-58.(in Chinese)
王元炜,郁梅,姜浩,等.一种自适应四叉树分块的图像拼接算法[J].宁波大学学报(理工版),2018,31(4):52-58.
[5]HE C,ZHOU J.Mesh-based image stitching algorithm with li- near structure protection[J].Journal of Image and Graphics,2018,23(7):0973-0983.(in Chinese)
何川,周军.具有直线结构保护的网格化图像拼接[J].中国图象图形学报,2018,23(7):973-983.
[6]ZHOU X,CAO S,HE X J,et al.Image stitching based on the planar similarity among matching pairs of feature points [J].Journal of University Electronic Science and Technology of China,2017,46(6):877-882.(in Chinese)
周雪,曹爽,何香静,等.基于特征点匹配对平面相似度的图像拼接[J].电子科技大学学报,2017,46(6):877-882.
[7]WANG F B,TU P,WU C,et al.Multi-image mosaic with SIFT and vision measurement for microscale structures processed by femtosecond laser [J].Optics and Lasers in Engineering,2018,100:124-130.
[8]FATHIMA A A,KARTHIK R,VAIDEHI V.Image stitching with combined moment invariants and sift features [J].Procedia Computer Science,2013,19:420-427.
[9]LIU W L,WANG Z Y,QING L B,et al.Application of LBP algorithm in rock slice image stitching[J].Computer & Digital Engineering,2016,44(2):326-330.(in Chinese)
刘文亮,王正勇,卿粼波,等.LBP 算法在岩石薄片图像拼接中的应用[J].计算机与数字工程,2016,44(2):326-330.
[10]CHEN Y,ZHAO Y,WANG S G.Fast image stitching method based on SIFT feature vector image [J].Journal of Jilin University (Science Edition),2017,5(1):116-122.(in Chinese)
陈月,赵岩,王世刚.基于 SIFT 特征矢量图的快速图像拼接方法[J].吉林大学学报(理学版),2017,55(1):116-122.
[11]LU J M,ZHU Z.Real-time 4K panoramic video stitching based on GPU acceleration [J].Computer Science,2017,44(8):18-21.(in Chinese)
卢嘉铭,朱哲.基于 GPU 加速的实时 4K 全景视频拼接[J].计算机科学,2017,44(8):18-21.
[12]LU Y Y,ZHANG M.Improved Algorithm Based on SIFT Infrared Image Stitching Algorithm [J].Computer Systems & Applications,2015,24(8):165-170.(in Chinese)
陆园园,张明.基于 SIFT 算法的红外图像拼接方法改进[J].计算机系统应用,2015,24(8):165-170.
[13]CHEN Y,ZHAO Y,WANG S G.Fast image stitching method based on SIFT with adaptive local image feature[J].Chinese Optics,2016,9(4):415-422.(in Chinese)
陈月,赵岩,王世刚.图像局部特征自适应的快速SIFT图像拼接方法[J].中国光学,2016,9(4):415-422.
[14]ZHAO Y,CHEN Y,WANG S G.Corrected fast SIFT image stitching method by combining projection error [J].Optics and Precision Engineering,2017,25(6):1645-1651.(in Chinese)
赵岩,陈月,王世刚.结合投影误差校正的快速 SIFT 图像拼接[J].光学精密工程,2017,25(6):1645-1651.
[15]XU W,MULLIGAN J.Performance evaluation of color correction approaches for automatic multi-view image and video stitching[C]∥2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).IEEE,2010:263-270.
[16]DU C,YUAN J,DONG J,et al.GPU based Parallel Optimization for Real Time Panoramic Video Stitching [J].arXiv1810.03988,2018.
[17]JIANG W,GU J.Video stitching with spatial-temporal content-preserving warping[C]∥Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.IEEE,2015:42-48.
[18]LIU J,LI S Y,LI R F.Multi-view Video Stitching of Outdoor Scenes [J].Computer Engineering,2016,42(4):259-265.(in Chinese)
刘娟,李实英,李仁发.室外场景的多视角视频拼接[J].计算机工程,2016,42(4):259-265.
[19]LI Y,DU B X.Real-time video splicing technology based on small region fusion [J].Journal of Jilin University (Science Edition),2016,54(6):1367-1372.(in Chinese)
李勇,杜丙新.基于小区域融合的实时视频拼接技术[J].吉林大学学报 (理学版),2016,54(6):1367-1372.
[20]YANG X P,HU Y,ZHANG K.Research on video mosaicking technology based on FPGA [J].Journal of Jilin University (Information Science Edition),2016,34(6):709-715.(in Chinese)
杨晓萍,胡玉,张凯.基于 FPGA 的视频拼接技术研究[J].吉林大学学报(信息科学版),2016,34(6):709-715.
[21]CHANG J Y,QIN R,LI Q,et al.Image quality assessment of panoramic image[J].Computer Science,2014,41(6):278-281.(in Chinese)
常嘉义,秦瑞,李庆,等.全景鸟瞰拼接图像的质量评价方法[J].计算机科学,2014,41(6):278-281.
[22]NOWOZIN S.Autopano-Sift,making panoramas fun[OL].ht- tp://user.cs.tu-berlin.De/nowozin/autopano-sift.
[23]PERBET F, JOHNSON S,PHAM M T,et al.Human Body Shape Estimation Using a Multi-resolution Manifold Forest[C]∥2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Columbus,OH,USA,2014:668-675.
[24]WANG J,WANG J D,ZENG G,et al.Fast Neighborhood Graph Search Using Cartesian Concatenation[C]∥2013 IEEE International Conference on Computer Vision (ICCV).Sydney,NSW,Australia,2013:2128-2135.

相关文章 15

[1]	刘云, 董守杰. 基于CUDA核函数的多路视频图像拼接加速算法 Acceleration Algorithm of Multi-channel Video Image Stitching Based on CUDA Kernel Function 计算机科学, 2022, 49(6A): 441-446. https://doi.org/10.11896/jsjkx.210600043
[2]	吕小敬, 刘钊, 褚学森, 石树鹏, 孟虹松, 黄震春. 面向超大规模并行模拟的LBM计算流体力学软件 Extreme-scale Simulation Based LBM Computing Fluid Dynamics Simulations 计算机科学, 2020, 47(4): 13-17. https://doi.org/10.11896/jsjkx.191000010
[3]	魏霖静, 宁璐璐, 郭斌, 侯振兴, 甘诗润. 基于混合蛙跳算法的K-mediods聚类挖掘与并行优化 K-mediods Cluster Mining and Parallel Optimization Based on Shuffled Frog Leaping Algorithm 计算机科学, 2020, 47(10): 126-129. https://doi.org/10.11896/jsjkx.190900113
[4]	胡志军,徐勇. 基于内容的视频检索综述 Overview of Content-based Video Retrieval 计算机科学, 2020, 47(1): 117-123. https://doi.org/10.11896/jsjkx.190100231
[5]	刘玉成, 理查德·丁, 张颖超. 一种BPNNs识别算法的医学检测泛实时性问题研究 Research on Pan-real-time Problem of Medical Detection Based on BPNNs Recognition Algorithm 计算机科学, 2018, 45(6): 301-307. https://doi.org/10.11896/j.issn.1002-137X.2018.06.053
[6]	郭鑫鹏,黄元元,胡作进. 基于关键帧的连续手语语句识别算法研究 Research on Continuous Sign Language Sentence Recognition Algorithm Based on Key Frame 计算机科学, 2017, 44(Z11): 178-183. https://doi.org/10.11896/j.issn.1002-137X.2017.11A.037
[7]	梁文乐,黄元元,胡作进. 基于二级匹配策略的实时动态手语识别 Real-time Dynamic Sign Language Recognition Based on Hierarchical Matching Strategy 计算机科学, 2017, 44(7): 299-303. https://doi.org/10.11896/j.issn.1002-137X.2017.07.054
[8]	姜文超,林穗,王多强,李东明,金海. Calculix三级并行优化及其在天河二号超级计算机中的应用 Three-level Parallel Optimization and Application of Calculix in TH-2 Super-computing Environments 计算机科学, 2017, 44(3): 32-35. https://doi.org/10.11896/j.issn.1002-137X.2017.03.008
[9]	钟忺,杨光,卢炎生. 基于双阈值滑动窗口子镜头分割和完全连通图的关键帧提取方法 Method of Key Frames Extraction Based on Double-threshold Values Sliding Window Sub-shot Segmentation and Fully Connected Graph 计算机科学, 2016, 43(6): 289-293. https://doi.org/10.11896/j.issn.1002-137X.2016.06.057
[10]	蒋勇,张海涛. 一种视频数据代表选择框架方法 Representative Selection Framework Approach for Videos 计算机科学, 2016, 43(11): 19-23. https://doi.org/10.11896/j.issn.1002-137X.2016.11.004
[11]	刘华咏,李涛. 基于改进分块颜色特征和二次提取的关键帧提取算法 Key Frame Extraction Algorithm Based on Improved Block Color Features and Second Extraction 计算机科学, 2015, 42(12): 307-311.
[12]	顾益军,解易,夏天. 基于内容代表性评价的关键帧抽取 Keyframe Extraction Based on Representative Evaluation of Contents 计算机科学, 2014, 41(8): 286-288. https://doi.org/10.11896/j.issn.1002-137X.2014.08.060
[13]	马正华,顾苏杭,戎海龙. 基于SIFT特征匹配的CamShift运动目标跟踪算法 CamShift Moving Object Tracking Algorithm Based on SIFT Feature Points Matching 计算机科学, 2014, 41(6): 291-294. https://doi.org/10.11896/j.issn.1002-137X.2014.06.058
[14]	郭延明,谢毓湘,老松杨,白亮. 相似视频片段的检测与定位方法研究 Detection and Location of Near-duplicate Video Clips 计算机科学, 2014, 41(10): 53-56. https://doi.org/10.11896/j.issn.1002-137X.2014.10.012
[15]	瞿中，高腾飞，张庆庆. 一种改进的视频关键帧提取算法研究 Study on an Improved Algorithm of Video Keyframe Extraction 计算机科学, 2012, 39(8): 300-303.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed