Computer Science ›› 2025, Vol. 52 ›› Issue (12): 141-149.doi: 10.11896/jsjkx.250400075

• Computer Graphics & Multimedia • Previous Articles     Next Articles

Appearance Enhancement and Semantic Segmentation-based Neural Radiance Fields

CAO Mingwei, HUANG Baolong, ZHAO Haifeng   

  1. School of Computer Science and Technology, Anhui University, Hefei 230601, China
  • Received:2025-04-15 Revised:2025-08-27 Online:2025-12-15 Published:2025-12-09
  • About author:CAO Mingwei,born in 1986,Ph.D,associate professor,master supervisor,is a member of CCF(No.49221M).His main research interests include 3D reconstruction and computer vision.
  • Supported by:
    This work was supported by the Anhui Province University Research Project(2024AH050045) and National Natural Science Foundation of China(62372153).

Abstract: The accelerated advancement of deep learning has notably propelled 3D reconstruction techniques within the field of computer vision.NeRFs have become an essential methodology due to their adeptness at scene modeling and superior view synthesis.However,challenges persist in dynamic environments,particularly in managing intricate lighting variations and transient object interference.Alterations in imaging conditions may lead to inconsistent scene appearances,thereby degrading the quality of view synthesis.Concurrently,dynamic elements can adversely affect the photorealism of reconstructed scenes.To mitigate these issues,this paper introduces an AS-NeRF.By incorporating frequency regularization and composite positional encoding into the sampling strategy,AS-NeRF enhances the efficiency of appearance feature fusion,thereby augmenting the model’s adaptability to variations in lighting and camera parameters.This subsequently improves color consistency and overall rendering realism.Additionally,a lightweight segmentation network is utilized to predict transient visibility masks in an end-to-end manner,effectively isolating dynamic objects and reducing their impact on view synthesis quality.The efficacy of AS-NeRF is verified through experiments conducted on the Photo Tourism datasets,which are compared qualitatively and quantitatively with several existingme-thods.The experimental results demonstrate that AS-NeRF surpasses existing approaches in terms of synthesis accuracy and further confirms the accuracy of the computed segmentation masks in distinguishing transient objects.

Key words: Neural radiance fields, View synthesis, Neural rendering, 3D reconstruction

CLC Number: 

  • TP391.41
[1]MILDENHALL B,SRINIVASAN P P,TANCIK M,et al.NeRF:Representing Scenes as Neural Radiance Fields for View Synthesis[J].Communications of the ACM,2021,65(1):99-106.
[2]MARTIN-BRUALLA R,RADWAN N,SAJJADI M S M,et al.NeRF in the Wild:Neural Radiance Fields for Unconstrained Photo Collections[C]//Proceedings of the IEEE/CVF Confe-rence on Computer Vision and Pattern Recognition.2021:7210-7219.
[3]CHEN X,ZHANG Q,LI X,et al.Hallucinated Neural Radiance Fields in the Wild[C]//Proceedings of the IEEE/CVF Confe-rence on Computer Vision and Pattern Recognition.2022:12943-12952.
[4]WANG S,XU H,LI Y,et al.IE-NeRF:Exploring TransientMask Inpainting to Enhance Neural Radiance Fields in the Wild[J].Neurocomputing,2025,618:129112.
[5]LEE J,KIM I,HEO H,et al.Semantic-Aware Occlusion Filtering Neural Radiance Fields in the Wild[J].arXiv:2303.03966,2023.
[6]SNAVELY N,SEITZ S,SZELISKI R.Photo Tourism:Explo-ring Photo Collections in 3D[J].ACM Transactions on Grap-hics,2006,25(3):835-846.
[7]CHAURASIA G,DUCHENE S,SORKINE-HORNUNG O,et al.Depth Synthesis and Local Warps for Plausible Image-Based Navigation[J].ACM Transactions on Graphics,2013,32(3):1-12.
[8]FITZGIBBON A,WEXLER Y,ZISSERMAN A.Image-BasedRendering Using Image-Based Priors[J].International Journal of Computer Vision,2005,63:141-151.
[9]LEVOY M,HANRAHAN P.Light Field Rendering [C]//Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques.1996:31-42.
[10]CHENG C M,LIN S J,LAI S H,et al.Improved Novel View Synthesis from Depth Image with Large Baseline[C]//Procee-dings of the 19th International Conference on Pattern Recognition.IEEE,2008:1-4.
[11]DEBEVEC P E,TAYLOR C J,MALIK J.Modeling and Rendering Architecture from Photographs:A Hybrid Geometry-and Image-Based Approach [C]//Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques.1996:11-20.
[12]BUEHLER C,BOSSE M,MCMILLAN L,et al.Unstructured Lumigraph Rendering[C]//Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques.2001:425-432.
[13]RIEGLER G,KOLTUN V.Free View Synthesis[C]//Procee-dings of the 16th European Conference Computer Vision.2020:623-640.
[14]FLYNN J,NEULANDER I,PHILBIN J,et al.Deepstereo:Learning to Predict New Views from The World’s Imagery[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2016:5515-5524.
[15]GROSSMAN J P,DALLY W J.Point Sample Rendering[C]//Rendering Techniques’ 98:Proceedings of the Eurographics Workshop in Vienna.1998:181-192.
[16]ZWICKER M,PFISTER H,VAN BAAR J,et al.Surface Splatting[C]//Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques.2001:371-378.
[17]DEBEVEC P,YU Y,BORSHUKOV G.Efficient View-Depen-dent Image-Based Rendering with Projective Texture-Mapping[C]//Rendering Techniques’ 98:Proceedings of the Eurogra-phics Workshop in Vienna.1998:105-116.
[18] SHAN Q,ADAMS R,CURLESS B,et al.The Visual TuringTest for Scene Reconstruction[C]//Proceedings of the IEEE International Conference on 3D Vision.2013:25-32.
[19]KUTULAKOS K N,SEITZ S M.A Theory of Shape by Space Carving[J].International Journal of Computer Vision,2000,38:199-218.
[20]PENNER E,ZHANG L.Soft 3d Reconstruction for View Synthesis[J].ACM Transactions on Graphics,2017,36(6):1-11.
[21]AKENINE-MOLLER T,HAINES E,HOFFMAN N.Real-Time Rendering[M].AK Peters/crc Press,2019.
[22]COOK R L,PORTER T,CARPENTER L.Distributed RayTracing[C]//Proceedings of the 11th Annual Conference on Computer Graphics and Interactive Techniques.1984:137-145.
[23]PURCELL T J,BUCK I,MARK W R,et al.Ray Tracing on Programmable Graphics Hardware[J].ACM Transactions on Graphics,2002,21(3):703-712.
[24]TATARCHENKO M,DOSOVITSKIY A,BROX T.Multi-View 3d Models from Single Images With a Convolutional Network[C]//Proceedings of the 14th European Conference on Computer Vision.2016:322-337.
[25]HEDMAN P,PHILIP J,PRICE T,et al.Deep Blending forFree-Viewpoint Image-Based Rendering[J].ACM Transactions on Graphics,2018,37(6):1-15.
[26]KO J,CHO K,CHOI D,et al.3d Gan Inversion with Pose Optimization[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision.2023:2967-2976.
[27]LIU L,GU J,LIN K Z,et al.Neural Sparse Voxel Fields[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems.2020:15651-15663.
[28]MÜLLER T,EVANS A,SCHIED C,et al.Instant NeuralGraphics Primitives with A Multiresolution Hash Encoding[J].ACM Transactions on Graphics,2022,41(4):1-15.
[29]FRIDOVICH-KEIL S,YU A,TANCIK M,et al.Plenoxels:Radiance Fields without Neural Networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2022:5501-5510.
[30]SUCAR E,LIU S,ORTIZ J,et al.IMAP:Implicit Mapping and Positioning in Real-Time[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.2021:6229-6238.
[31]SUN C,SUN M,CHEN H T.Direct Voxel Grid Optimization:Super-Fast Convergence for Radiance Fields Reconstruction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2022:5459-5469.
[32]CHIBANE J,BANSAL A,LAZOVA V,et al.Stereo Radiance Fields(Srf):Learning View Synthesis for Sparse Views of Novel Scenes[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:7911-7920.
[33]CHEN A,XU Z,ZHAO F,et al.MVSNeRF:Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.2021:14124-14133.
[34]YU A,YE V,TANCIK M,et al.PixelNeRF:Neural Radiance Fields from One or Few Images[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:4578-4587.
[35]KIM M,SEO S,HAN B.InfoNeRF:Ray Entropy Minimization for Few-Shot Neural Volume Rendering[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2022:12912-12921.
[36]WANG Z,BAGAUTDINOV T,LOMBARDI S,et al.Learning Compositional Radiance Fields of Dynamic Human Heads[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:5704-5713.
[37]WANG X B,ZHANG H,GAO H,et al.Talking Portrait Synthesis Method Based on Regional Saliency and Spatial Feature Extraction [J].Computer Science,2025,52(3):58-67.
[38]PARK J J,FLORENCE P,STRAUB J,et al.DeepSDF:Learning Continuous Signed Distance Functions for Shape Representation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2019:165-174.
[39]MESCHEDER L,OECHSLE M,NIEMEYER M,et al.Occu-pancy Networks:Learning 3d Reconstruction in Function Space[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2019:4460-4470.
[40]SAITO S,HUANG Z,NATSUME R,et al.Pifu:Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization[C]//Proceedings of the IEEE/CVF International Confe-rence on Computer Vision.2019:2304-2314.
[41]SITZMANN V,MARTEL J,BERGMAN A,et al.Implicit Neural Representations with Periodic Activation Functions[C]//Proceedings of the 33th International Conference on Neural Information Processing Systems(NeurlPS).2020:7462-7473.
[42]LOMBARDI S,SIMON T,SARAGIH J M,et al.Neural Volumes[J].ACM Transactions on Graphics,2019,38:1-14.
[43]BARRON J T,MILDENHALL B,VERBIN D,et al.Mip-NeRF 360:Unbounded Anti-Aliased Neural Radiance Fields[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2022:5470-5479.
[44]BARRON J T,MILDENHALL B,TANCIK M,et al.Mip-Nerf:A Multiscale Representation for Anti-Aliasing Neural Radiance Fields[C]//Proceedings of the IEEE/CVF International Confe-rence on Computer Vision.2021:5855-5864.
[45]WU T,TANG S,ZHANG R,et al.CGNet:A Light-WeightContext Guided Network for Semantic Segmentation[J].IEEE Transactions on Image Processing,2020,30:1169-1179.
[46]MAO Q,LEE H Y,TSENG H Y,et al.Mode Seeking Generative Adversarial Networks for Diverse Image Synthesis[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2019:1429-1437.
[1] SONG Xingnuo, WANG Congyan, CHEN Mingkai. Survey on 3D Scene Reconstruction Techniques in Metaverse [J]. Computer Science, 2025, 52(3): 17-32.
[2] WANG Jie, WANG Chuangye, XIE Jiucheng, GAO Hao. Animatable Head Avatar Reconstruction Algorithm Based on Region Encoding [J]. Computer Science, 2025, 52(3): 50-57.
[3] WANG Xingbo, ZHANG Hao, GAO Hao, ZHAI Mingliang, XIE Jiucheng. Talking Portrait Synthesis Method Based on Regional Saliency and Spatial Feature Extraction [J]. Computer Science, 2025, 52(3): 58-67.
[4] ZHONG Yue, GU Jieming. 3D Reconstruction of Single-view Sketches Based on Attention Mechanism and Contrastive Loss [J]. Computer Science, 2025, 52(3): 77-85.
[5] LI Pengfei, GUAN Xiancai, ZHU Youjian, LI Yuanqiao, WANG Jun. Optimization and Absolute Scale Recovery of SFM Algorithm in GCP-assisted Colmap Framework [J]. Computer Science, 2025, 52(11A): 250100015-6.
[6] YE Ruiwen, WANG Baohui. 3D Reconstruction Algorithm for Lower Limb X-ray Images Based on Generative AdversarialNetworks [J]. Computer Science, 2024, 51(11A): 230900089-7.
[7] HE Weilong, SU Lingli, GUO Bingxuan, LI Maosen, HAO Yan. Research and Implementation of Dynamic Scene 3D Perception Technology Based on BinocularEstimation [J]. Computer Science, 2024, 51(11A): 240300045-8.
[8] ZHANG Xiao, DONG Hongbin. Lightweight Multi-view Stereo Integrating Coarse Cost Volume and Bilateral Grid [J]. Computer Science, 2023, 50(8): 125-132.
[9] FENG Lei, ZHU Deng-ming, LI Zhao-xin, WANG Zhao-qi. Sparse Point Cloud Filtering Algorithm Based on Mask [J]. Computer Science, 2022, 49(5): 25-32.
[10] REN Fei, CHANG Qing-ling, LIU Xing-lin, YANG Xin, LI Ming-hua, CUI Yan. Overview of 3D Reconstruction of Indoor Structures Based on Point Clouds [J]. Computer Science, 2022, 49(11A): 211000176-11.
[11] MA Jun-cheng, JIANG Mu-rong, FANG Su-qin. Three-dimensional Reconstruction of Cone Meteorological Data Based on Improved MarchingTetrahedra Algorithm [J]. Computer Science, 2021, 48(11A): 644-647.
[12] ZENG Jun-fei,YANG Hai-qing,WU Hao. Adaptive Levenberg-Marquardt Cloud Registration Method for 3D Reconstruction [J]. Computer Science, 2020, 47(3): 137-142.
[13] MIAO Hui-cui, WANG Ji-hua, ZHANG Quan-ying. Concave-convex Manufacturing Features Recognition Based on 3D Reconstruction of Single View [J]. Computer Science, 2019, 46(7): 280-285.
[14] MENG Wen-quan, WU Li-sheng. Triangulation Reconstruction of Plantar Surface Based on Depth Feature [J]. Computer Science, 2019, 46(6A): 205-207.
[15] CHEN Xi, LI Lei-da, LI Qiao-yue, HAN Xi-xi, ZHU Han-cheng. No-reference Quality Assessment of Depth Images Based on Natural Scenes Statistics [J]. Computer Science, 2019, 46(6): 256-262.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!