Computer Science

Computer Science ›› 2023, Vol. 50 ›› Issue (11A): 230300146-9.doi: 10.11896/jsjkx.230300146

• Computer Software & Architecture • Previous Articles     Next Articles

Large-scale Efficient Hybrid Parallel Computing for DSMC/PIC Coupled Simulation

WANG Qingsong1, QIU Haozhong1, LIN Yongzhen1, YANG Fuxiang2,3, LI Jie2, WANG Zhenghua4, XU Chuanfu1   

  1. 1 Institute for Quantum Information & State Key Laboratory of High Performance Computing, National University of Defense Technology, Changsha 410000, China
    2 College of Aerospace and Engineering,National University of Defense Technology,Changsha 410000,China
    3 Army Military Transportation University,Bengbu,Anhui 233000,China
    4 College of Computer,National University of Defense Technology,Changsha 410000,China
  • Published:2023-11-09
  • About author:WANG Qingsong,born in 1999,postgraduate.His main research interests in high-performance computing applications.
    XU Chuanfu,born in 1980,Ph.D,associate researcher,master supervisor.His main research interests include parallel computing and large-scale science and engineering computing.
  • Supported by:
    National Numerical Windtunnel Project(TC228S03J) and Sichuan Science and Technology Program(2023YFG0152).

PDF (PC)

657

Abstract: DSMC/PIC coupled simulation is an important class of high-performance computing applications.Due to the dynamic particle injection and migration,the pure MPI parallelization of DSMC/PIC coupled simulation usually suffers from huge communications costs and load imbalance.In this paper,we present approaches to implement large-scale and efficient MPI+OpenMP hybrid parallelization and dynamic load balancing research for a self-developed DSMC/PIC coupled simulation software.Firstly,we propose a MPI parallel algorithm based on nested dual unstructured grid with two parallel communication strategies,centralized and distributed,to support the dynamic migration of particles between any parallel processes.Then,we present a weighted load performance model,and a dynamic load balancing algorithm and an efficient grid remapping mechanism are designed and implemented,which greatly improves the parallel efficiency of coupled parallel simulation.Furthermore,we design and implement a hybrid parallel algorithm of MPI+OpenMP for coupled simulation,which effectively reduces the grid redecomposition and communication overheads of pure MPI parallelization with dynamic load balance.On the BSCC HPC system,the DSMC/PIC coupled parallel simulation of thousands of processor cores is carried out for the billion particle scale pulsed vacuum arc plasma plume,and the effect of the parallel algorithm and dynamic load balancing has been verified.

Key words: Coupled DSMC/PIC, Particle simulation, Centralized and distributed communication strategies, Dynamic load balance, MPI+OpenMP

CLC Number: 

  • TP391
[1]BIRD G A.Molecular Gas Dynamics and the Direct Simulation of Gas Flows[M].Oxford,1976.
[2]KBIRDSALL C,LANGDON A B.Plasma Physics Via Computer Simulation[J].Computer Physics Communications,1986,42:151-152.
[3]WHOCKNEY R,EASTWOOD J W.Computer simulationusing particles[J].Institute of Physics,1988,76.
[4]BIRD G A.Direct simulation of the boltzmann equation[J].Physics of Fluids,1970,13(11):2676-2681.
[5]COPPLESTONE S,ORTWEIN P,MUNZ C D,et al.Coupled PIC-DSMC Simulations of a Laser-Driven Plasma Expansion[M]//High Performance Computing in Science and Enginee-ring ’15.Springer International Publishing,2016,15:689-701.
[6]COPPLESTONE S,MUNZ C D,PFEIFFER M.PIC-DSMCsimulations of plasma plume expansions with ionization and recombination processes[C]//IEEE International Conference on Plasma Science.IEEE,2016:1-1.
[7]SMITH B D,BOYD I D,KAMHAWI H,et al.Hybrid-picmodeling of a high-voltage,high-specific-impulse hall thruster[C]//AIAA/ASME/SAE/ASEE.2013.
[8]KORKUT B,LI Z,LEVIN D A.3-d simulation of ion thruster plumesusing octree adaptive mesh refinement[J].IEEE Transactions on Plasma Science,2015,43(5):1706-1721.
[9]BRIEDA L,TAI S Z,KEIDAR M.Near plume modeling of amicro cathode arc thruster[C]//AIAA/ASME/SAEE/ASEE Joint Propulsion Conference.2013.
[10]TACCOGNA F,MINELLI P,BRUNO D,et al.Kineticdivertor modeling[J].Chemical Physics,2012,398(none):27-32.
[11]GLEASON-GONZALEZ C,VAROUTIS S,HAUER V,et al.Simulationof neutral gas flow in a tokamak divertor using the direct simulationmonte carlo method[J].Fusion Engineering & Design,2014,89(7/8):1042-1047.
[12]XU C,ZHANG L,DENG X,et al.Balancing cpu-gpu collaborative high-order cfd simulations on the tianhe-1a supercomputer[C]//IEEE International Parallel & Distributed Processing Symposium.2014.
[13]XU C,DENG X,ZHANG L,et al.Collaborating cpu and gpu for largescale high-order cfd simulations with complex grids on the tianhe-1a supercomputer[J].Journal of Computational Physics,2014,278:275-297.
[14]HYA W.The hungarian method for the assignment problem[J].Naval Research Logistics,1955,2:83-97.
[15]MUNKRES J.Algorithms for the Assignment and Transportation Problems[J].Journal of the Society for Industrial and Applied Mathematics,1957,5(1):32-38.
[16]HOPKINS,MATTHEWM,BOERNER,et al.Addressing challenges to simulating breakdown and arcevolution in vacuum and low pressure systems[C]//International Conference on Numerical Simulation of Plasmas.2013.
[17]MHOPKINS M,BOERNER J J,MOORE C H,et al.Ppps-2013:Accommodating large temporal,spatial,andparticle weighting demands for simulating vacuum arc discharge[C]//Abstracts IEEE International Conference on Plasma Science.2013.
[18]HOPKINS M M,MANGINELL R P,BOERNER J J,et al.Fully kinetic simulation of atmospheric pressuremicrocavity discharge device[C]//IEEE International Conference onPlasma Sciences.2015:1-1.
[19]ORTWEINP,BINDERT,COPPLESTONES,et al.Parallel per-formance of a discontinuous galerkin spectral elementmethod based pic-dsmc solver [C]//High Performance Computing in Science and Engineering’14:Transactions of the high performance computing center,STUTTGART(HLRS) 2014.2015:671-681.
[20]COPPLESTONES,ORTWEINP,MUNZC D,et al.Coupled pic-dsmc simulations of a laser-driven plasma expansion[C]//High Performance Computing in Science and Engineering ’15.2016:689-701.
[21]JOHNSON C,PITKGRANTA J.An analysis of the discontinuousgalerkin method for a scalar hyperbolic equation[J].Mathematics of Computation,1986,46(173):1-26.
[22]KORKUT B,LEVIN D A.Three dimensional dsmc-pic simulations ofion thruster plumes with sugar[C]//AIAA/ASME/SAEE/ASEE Joint Propulsion Conference.2014.
[23]REVATHIJ,LEVIND A.Chaos:An octree-based pic-dsmc code formodeling of electron kinetic properties in a plasma plume using mpicuda parallelization[J].Journal of Computational Phy-sics,2018,373:571-604.
[24]LI J,INGHAM D,MA L,et al.Numericalsimulation of thechemical combination and dissociation reactions ofneutral particles in a rarefied plasma arc jet[J].IEEE Transactions on Plasma Science,2017,45(3):461-471.
[25]YS U,LI J,WANG H,et al.Numerical simulation of chemicalreactions on rarefied plasma plume by dsmc method[J].IEEE Transactionson Plasma Science,2021,49(3):1-13.
[26]BIRD G A.Definition of mean free path for real gases[J].Physics of Fluids,1983,26(11):3222-3223.
[27]BIRD G A.Molecular gas dynamics and the direct simulation of gas flow[J].Clarendon Press,1994.
[28]BORISJ P.Relativistic plasma simulations-optimization of a hybridcode[C]//International Conference on Numerical Simulation of Plasmas.1970.
[29]LIMA E,TAVARES F W,JR E B.Finite volume solution ofthe modified poisson-boltzmann equation for two colloidal particles[J].Physical Chemistry Chemical Physics,2007,9(24):3174-3180.
[30]K L U.of Minnesota.Metis-serial graph partitioning andfill-reducing matrix ordering[OL].http://glaros.dtc.umn.edu /gkhome/metis/metis/overview,2013.
[31]LABORATORY A N.Petsc 3.16－petsc 3.16.0 documentation[OL].https://petsc.org/release/,2021.
[32]STRUMPACK:STRUctured Matrices PACKages[OL].http://portal.nersc.gov/project/sparse/strumpack/.
[1] LIU Jiang, LIU Wen-bo, ZHANG Ju. Hybrid MPI+OpenMP Parallel Method on Polyhedral Grid Generation in OpenFoam [J]. Computer Science, 2022, 49(3): 3-10.
[2] ZHOU Jie and LI Wen-jing. Research on Parallel Algorithm of Petri Net Based on Three-layer Mixed Programming Model [J]. Computer Science, 2017, 44(Z11): 586-591.
[3] LIU Xu, MO Ze-yao, AN Heng-bin, CAO Xiao-lin and ZHANG Ai-qing. Automatic Load Modeling Algorithm Based on Real Time Measuring [J]. Computer Science, 2015, 42(1): 63-66.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
Add:18# Honghu West Rd., Yubei ,Chongqing,China    Post Code: 401121
Tel: 023-63500828/023-67039612/023-67039623
E-mail: jsjkx12@163.com
Powered by Beijing Magtech Co., Ltd.