计算机科学

计算机科学 ›› 2023, Vol. 50 ›› Issue (11A): 230300146-9.doi: 10.11896/jsjkx.230300146

• 计算机软件&体系架构 • 上一篇    下一篇

DSMC/PIC耦合模拟的大规模高效混合并行计算研究

汪青松1, 邱昊中1, 林拥真1, 杨富翔2,3, 李洁2, 王正华4, 徐传福1   

  1. 1 国防科技大学计算机学院量子信息研究所兼高性能计算国家重点实验室 长沙 410000
    2 国防科技大学空天科学学院 长沙 410000
    3 军事交通学院 安徽 蚌埠233000
    4 国防科技大学计算机学院 长沙 410000
  • 发布日期:2023-11-09
  • 通讯作者: 徐传福(xuchuanfu@nudt.edu.cn)
  • 作者简介:(wangqs@nudt.edu.cn)
  • 基金资助:
    国家数值风洞工程(TC228S03J);四川省科技计划(2023YFG0152)

Large-scale Efficient Hybrid Parallel Computing for DSMC/PIC Coupled Simulation

WANG Qingsong1, QIU Haozhong1, LIN Yongzhen1, YANG Fuxiang2,3, LI Jie2, WANG Zhenghua4, XU Chuanfu1   

  1. 1 Institute for Quantum Information & State Key Laboratory of High Performance Computing, National University of Defense Technology, Changsha 410000, China
    2 College of Aerospace and Engineering,National University of Defense Technology,Changsha 410000,China
    3 Army Military Transportation University,Bengbu,Anhui 233000,China
    4 College of Computer,National University of Defense Technology,Changsha 410000,China
  • Published:2023-11-09
  • About author:WANG Qingsong,born in 1999,postgraduate.His main research interests in high-performance computing applications.
    XU Chuanfu,born in 1980,Ph.D,associate researcher,master supervisor.His main research interests include parallel computing and large-scale science and engineering computing.
  • Supported by:
    National Numerical Windtunnel Project(TC228S03J) and Sichuan Science and Technology Program(2023YFG0152).

PDF (PC)

172

摘要： DSMC/PIC耦合模拟是一类重要的高性能计算应用。由于粒子动态注入、迁移等操作,传统MPI并行DSMC/PIC耦合模拟通常并行通信开销较大且负载不均衡。文中针对自主研发的DSMC/PIC耦合模拟软件,开展了大规模高效MPI+OpenMP混合并行及动态负载均衡研究。首先设计了基于嵌套双重非结构网格的MPI并行算法,实现了集中式和分布式两种并行通信策略,支持粒子在任意并行进程间的动态迁移。然后提出了加权负载性能模型,设计了动态负载均衡算法及高效网格重映射机制,大幅提升了耦合模拟并行效率,进一步设计了MPI+OpenMP混合并行算法,有效降低了纯MPI并行计算中动态负载均衡的网格重剖分和通信开销。在北京北龙超级云HPC系统上,针对10亿粒子规模脉冲真空弧等离子体羽流开展了数千处理器核心DSMC/PIC耦合并行模拟,验证了并行算法和动态负载均衡的效果。

关键词: DSMC/PIC耦合, 粒子模拟, 分布式和集中式, 动态负载均衡, MPI+OpenMP

Abstract: DSMC/PIC coupled simulation is an important class of high-performance computing applications.Due to the dynamic particle injection and migration,the pure MPI parallelization of DSMC/PIC coupled simulation usually suffers from huge communications costs and load imbalance.In this paper,we present approaches to implement large-scale and efficient MPI+OpenMP hybrid parallelization and dynamic load balancing research for a self-developed DSMC/PIC coupled simulation software.Firstly,we propose a MPI parallel algorithm based on nested dual unstructured grid with two parallel communication strategies,centralized and distributed,to support the dynamic migration of particles between any parallel processes.Then,we present a weighted load performance model,and a dynamic load balancing algorithm and an efficient grid remapping mechanism are designed and implemented,which greatly improves the parallel efficiency of coupled parallel simulation.Furthermore,we design and implement a hybrid parallel algorithm of MPI+OpenMP for coupled simulation,which effectively reduces the grid redecomposition and communication overheads of pure MPI parallelization with dynamic load balance.On the BSCC HPC system,the DSMC/PIC coupled parallel simulation of thousands of processor cores is carried out for the billion particle scale pulsed vacuum arc plasma plume,and the effect of the parallel algorithm and dynamic load balancing has been verified.

Key words: Coupled DSMC/PIC, Particle simulation, Centralized and distributed communication strategies, Dynamic load balance, MPI+OpenMP

中图分类号: 

  • TP391
[1]BIRD G A.Molecular Gas Dynamics and the Direct Simulation of Gas Flows[M].Oxford,1976.
[2]KBIRDSALL C,LANGDON A B.Plasma Physics Via Computer Simulation[J].Computer Physics Communications,1986,42:151-152.
[3]WHOCKNEY R,EASTWOOD J W.Computer simulationusing particles[J].Institute of Physics,1988,76.
[4]BIRD G A.Direct simulation of the boltzmann equation[J].Physics of Fluids,1970,13(11):2676-2681.
[5]COPPLESTONE S,ORTWEIN P,MUNZ C D,et al.Coupled PIC-DSMC Simulations of a Laser-Driven Plasma Expansion[M]//High Performance Computing in Science and Enginee-ring ’15.Springer International Publishing,2016,15:689-701.
[6]COPPLESTONE S,MUNZ C D,PFEIFFER M.PIC-DSMCsimulations of plasma plume expansions with ionization and recombination processes[C]//IEEE International Conference on Plasma Science.IEEE,2016:1-1.
[7]SMITH B D,BOYD I D,KAMHAWI H,et al.Hybrid-picmodeling of a high-voltage,high-specific-impulse hall thruster[C]//AIAA/ASME/SAE/ASEE.2013.
[8]KORKUT B,LI Z,LEVIN D A.3-d simulation of ion thruster plumesusing octree adaptive mesh refinement[J].IEEE Transactions on Plasma Science,2015,43(5):1706-1721.
[9]BRIEDA L,TAI S Z,KEIDAR M.Near plume modeling of amicro cathode arc thruster[C]//AIAA/ASME/SAEE/ASEE Joint Propulsion Conference.2013.
[10]TACCOGNA F,MINELLI P,BRUNO D,et al.Kineticdivertor modeling[J].Chemical Physics,2012,398(none):27-32.
[11]GLEASON-GONZALEZ C,VAROUTIS S,HAUER V,et al.Simulationof neutral gas flow in a tokamak divertor using the direct simulationmonte carlo method[J].Fusion Engineering & Design,2014,89(7/8):1042-1047.
[12]XU C,ZHANG L,DENG X,et al.Balancing cpu-gpu collaborative high-order cfd simulations on the tianhe-1a supercomputer[C]//IEEE International Parallel & Distributed Processing Symposium.2014.
[13]XU C,DENG X,ZHANG L,et al.Collaborating cpu and gpu for largescale high-order cfd simulations with complex grids on the tianhe-1a supercomputer[J].Journal of Computational Physics,2014,278:275-297.
[14]HYA W.The hungarian method for the assignment problem[J].Naval Research Logistics,1955,2:83-97.
[15]MUNKRES J.Algorithms for the Assignment and Transportation Problems[J].Journal of the Society for Industrial and Applied Mathematics,1957,5(1):32-38.
[16]HOPKINS,MATTHEWM,BOERNER,et al.Addressing challenges to simulating breakdown and arcevolution in vacuum and low pressure systems[C]//International Conference on Numerical Simulation of Plasmas.2013.
[17]MHOPKINS M,BOERNER J J,MOORE C H,et al.Ppps-2013:Accommodating large temporal,spatial,andparticle weighting demands for simulating vacuum arc discharge[C]//Abstracts IEEE International Conference on Plasma Science.2013.
[18]HOPKINS M M,MANGINELL R P,BOERNER J J,et al.Fully kinetic simulation of atmospheric pressuremicrocavity discharge device[C]//IEEE International Conference onPlasma Sciences.2015:1-1.
[19]ORTWEINP,BINDERT,COPPLESTONES,et al.Parallel per-formance of a discontinuous galerkin spectral elementmethod based pic-dsmc solver [C]//High Performance Computing in Science and Engineering’14:Transactions of the high performance computing center,STUTTGART(HLRS) 2014.2015:671-681.
[20]COPPLESTONES,ORTWEINP,MUNZC D,et al.Coupled pic-dsmc simulations of a laser-driven plasma expansion[C]//High Performance Computing in Science and Engineering ’15.2016:689-701.
[21]JOHNSON C,PITKGRANTA J.An analysis of the discontinuousgalerkin method for a scalar hyperbolic equation[J].Mathematics of Computation,1986,46(173):1-26.
[22]KORKUT B,LEVIN D A.Three dimensional dsmc-pic simulations ofion thruster plumes with sugar[C]//AIAA/ASME/SAEE/ASEE Joint Propulsion Conference.2014.
[23]REVATHIJ,LEVIND A.Chaos:An octree-based pic-dsmc code formodeling of electron kinetic properties in a plasma plume using mpicuda parallelization[J].Journal of Computational Phy-sics,2018,373:571-604.
[24]LI J,INGHAM D,MA L,et al.Numericalsimulation of thechemical combination and dissociation reactions ofneutral particles in a rarefied plasma arc jet[J].IEEE Transactions on Plasma Science,2017,45(3):461-471.
[25]YS U,LI J,WANG H,et al.Numerical simulation of chemicalreactions on rarefied plasma plume by dsmc method[J].IEEE Transactionson Plasma Science,2021,49(3):1-13.
[26]BIRD G A.Definition of mean free path for real gases[J].Physics of Fluids,1983,26(11):3222-3223.
[27]BIRD G A.Molecular gas dynamics and the direct simulation of gas flow[J].Clarendon Press,1994.
[28]BORISJ P.Relativistic plasma simulations-optimization of a hybridcode[C]//International Conference on Numerical Simulation of Plasmas.1970.
[29]LIMA E,TAVARES F W,JR E B.Finite volume solution ofthe modified poisson-boltzmann equation for two colloidal particles[J].Physical Chemistry Chemical Physics,2007,9(24):3174-3180.
[30]K L U.of Minnesota.Metis-serial graph partitioning andfill-reducing matrix ordering[OL].http://glaros.dtc.umn.edu /gkhome/metis/metis/overview,2013.
[31]LABORATORY A N.Petsc 3.16－petsc 3.16.0 documentation[OL].https://petsc.org/release/,2021.
[32]STRUMPACK:STRUctured Matrices PACKages[OL].http://portal.nersc.gov/project/sparse/strumpack/.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
地址：重庆市渝北区洪湖西路18号    邮编：401121  
电话：023-63500828（编辑部） 023-67039612（会议/专题） 023-67039623（财务/发票）
E-mail：jsjkx12@163.com
本系统由北京玛格泰克科技发展有限公司设计开发