计算机科学 ›› 2021, Vol. 48 ›› Issue (1): 319-325.doi: 10.11896/jsjkx.191100111

• 交叉与前沿 • 上一篇    下一篇


徐蕴琪1,2, 黄荷1,3, 金钟1,3   

  1. 1 中国科学院计算机网络信息中心 北京 100190
    2 中国科学院大学 北京 100049
    3 中国科学院计算科学应用研究中心 北京 100190
  • 收稿日期:2019-11-14 修回日期:2020-03-24 出版日期:2021-01-15 发布日期:2021-01-15
  • 通讯作者: 黄荷(huanghe@sccas.cn)
  • 作者简介:jocelyn_1234@163.com
  • 基金资助:
    国家重点研发计划项目(2016YFB0201700);广东省生物医药计算重点实验室(2016B030301007);中国科学院科研信息化应用工程项目 (XXH13506-202)

Application Research on Container Technology in Scientific Computing

XU Yun-qi1,2, HUANG He1,3, JIN Zhong1,3   

  1. 1 Computer Network Information Center,Chinese Academy of Sciences,Beijing 100190,China
    2 University of Chinese Academy of Sciences,Beijing 100049,China
    3 Center of Scientific Computing Applications & Research,Chinese Academy of Sciences,Beijing 100190,China
  • Received:2019-11-14 Revised:2020-03-24 Online:2021-01-15 Published:2021-01-15
  • About author:XU Yun-qi,born in 1994,postgraduate.Her main research interests include scientific computing and application service development.
    HUANG He,born in 1982,Ph.D,associate professor.Her main research interests include scientific computing and data ser-vice.
  • Supported by:
    National Key Research and Development Project(2016YFB0201700),Guangdong Provincial Key Laboratory of Biocomputing(2016B030301007) and Information Program of the Chinese Academy of Sciences(XXH13506-202).

摘要: 作为一种新兴的虚拟化技术,容器能够以低廉的资源开销为应用程序和服务提供隔离的运行环境,近年来在持续集成和持续部署、自动化测试、微服务等多种业务场景中获得了广泛应用。在科学计算领域,容器技术的应用也正获得越来越多的关注。借助自身的打包能力及日益壮大的生态系统,容器技术有望为科学计算领域的生产力提升提供助力。文中对容器技术在科学计算中的应用现状进行了调研分析,并根据现有的应用实例讨论了在科学计算中使用容器及相关技术的多种方式。对不同应用模式的分析研究表明,通过提升应用程序的可移植性、改善研究的可重复性、提供非传统应用部署方案、简化云资源调度管理等多种方式,容器及相关技术可以为科学计算领域带来多方面的效率提升。

关键词: 科学计算, 容器, 容器编排框架, 云计算, 作业调度系统

Abstract: Container is a new virtualization technology that has emerged in recent years.Due to its ability to provide isolated environment for running applications and services with minimal resource overhead,it quickly explodes in popularity among enterprises and has seen wide applications in a number of business scenarios such as continuous integration,continuous deployment,automated testing and micro-services.Although not as fully utilized as in industry,the packaging ability of containers also holds promise for improving productivity and code portability in the domain of scientific computing.In this paper,we discuss how the container and related technologies can be used in scientific computing by surveying existing application examples.The different application patterns represented by these examples suggest that the scientific computing community may benefit from the container technology and the ecosystem evolving around it in many different ways.

Key words: Cloud computing, Container, Container orchestration framework, Job scheduling system, Scientific computing


  • TP391
[1] Cloud Developer.The Evolution of Container Technology[EB/OL].(2018-04-17) [2019-08-02].https://www.cnblogs.com/bakari/p/8868850.html.
[2] YANG B H,DAI W J,CAO Y L.Docker Primer[M].Beijing:China Machine Press,2015:3-6.
[3] MENAGE P B.Adding Generic Process Containers to the Linux Kernel[C]//Proceedings of the Linux Symposium.2007,2:45-57.
[4] WU Z X.Advances on Virtualization Technology of CloudComputing[J].Journal of Computer Applications,2017,37(4):915-923.
[5] WU S,WANG K,JIN H.Research Situation and Prospects ofOperating System Virtualization[J].Journal of Computer Research and Development,2019,56(1):58-68.
[6] BERNSTEIN D.Containers and Cloud:From LXC to Docker to Kubernetes[J].IEEE Cloud Computing,2014,1(3):81-84.
[7] HINDMAN B,KONWINSKI A,ZAHARIA M,et al.Mesos:APlatform for Fine-Grained Resource Sharing in the Data Center[C]//NSDI.2011,11(2011):22.
[8] Docker Inc.Docker Swarm Overview[EB/OL].(2018-02-24) [2019-08-06].https://docs.docker.com/swarm/overview/.
[9] LOU C.An Analysis of the Application Scenari-os of Docker Container[EB/OL].(2018-09-11) [2019-08-06].https://sq.163yun.com/blog/article/197408114326077440.
[10] ZHU S P.A Brief Report on Scientific Computing[J].Physics,2009(8):545-551.
[11] ZHANG X L,ZHONG Y P.A Performance Evaluation andCompare for Cluster-based Resource Management Systems[J].Application Research of Computers,2003(9):56-59.
[12] ZHOU S,ZHENG X,WANG J,et al.Utopia:A Load Sharing Facility for Large,Heterogeneous Distributed Computer Systems[J].Software:Practice and Experience,1993,23(12):1305-1336.
[13] YOO A B,JETTE M A,GRONDONA M.Slurm:Simple Linux Utility for Resource Management[C]//Workshop on Job Sche-duling Strategies for Parallel Processing.Springer,Berlin,Heidelberg,2003:44-60.
[14] NITZBERG B,SCHOPF J M,JONES J P.PBS Pro:Grid Computing and Scheduling Attributes[M]//Grid Resource Management.Springer,Boston,MA,2004:183-190.
[15] LEHTO O P.Containers,Meet HPC[EB/OL].(2015-12-30)[2019-08-12].https://medium.com/@ople/containers-meet-hpc-2aab7aa2d54a.
[16] TELFER S.The State of HPC Containers[EB/OL].(2018-08-16) [2019-08-12].https://www.stackhpc.com/the-state-of-hpc-containers.html.
[17] WANG J,HU W,ZHANG Y H,et al.Trusted Container Based on Docker[J].Journal of Wuhan University(Natural Science Edition),2017,63(2):102-108.
[18] MEDRANO-JAIMES F,LOZANO-RIZK J E,CASTAÑEDA-AVILA S,et al.Use of Containers for High-Performance Computing[C]//International Conference on Supercomputing in Mexico.Springer,Cham,2018:24-32.
[19] YOUNGE A J,PEDRETTI K,GRANT R E,et al.A Tale of Two Systems:Using Containers to Deploy HPC Applications on Supercomputers And Clouds[C]//2017 IEEE International Conference on Cloud Computing Technology and Science(CloudCom).IEEE,2017:74-81.
[20] GERHARDT L,BHIMJI W,FASEL M,et al.Shifter:Contai-ners for HPC[C]//J.Phys.Conf.Ser.2017,898:082021.
[21] JACOBSEN D M,CANON R S.Contain This,Unleashing Doc-ker for HPC[C]//Proceedings of the Cray User Group,2015:33-49.
[22] NERSC.How to Use Shifter[EB/OL].(2019-03-22) [2019-08-18].https://docs.nersc.gov/programming/shifter/how-to-use.
[23] PAN X.Shifter:Containers for HPC [EB/OL].(2017-08-23) [2019-08-18].https://www.ibm.com/developerworks/cn/linux/l-panxun-shifter-compute-container/index.html.
[24] KURTZER G M,SOCHAT V,BAUER M W.Singularity:Scientific Containers for Mobility of Compute[J].PLoS One,2017,12(5):e0177459.
[25] IDG Communications.Utilizing Containers for HPC and DeepLearning Workloads[EB/OL].(2018-05-17) [2019-08-19].https://www.cio.com/article/3269351/utilizing-containers-for-hpc-and-deep-learning-workloads.html.
[26] Singularity.Singularity Docs-Admin Guide[EB/OL].(2018-06-21) [2019-08-19].https://www.sylabs.io/guides/2.5/admin-guide/installation_environments.html.
[27] PRIEDHORSKY R,RANDLES T.Charliecloud:UnprivilegedContainers for User-defined Software Stacks in HPC[C]//Proceedings of the International Conference for High Performance Computing,Networking,Storage and Analysis.ACM,2017:36.
[28] DANDAPANTHULA N.Singularity Containers for HPC &Deep Learning[EB/OL].(2018-03-19) [2019-08-20].https://www.nextplatform.com/2018/03/19/singularity-containers-for-hpc-deep-learning.
[29] BAO X.Alibaba Cloud Supercomputing:Singularity for High-Performance Container Solutions[EB/OL].(2018-11-16) [2019-12-20].https://yq.aliyun.com/articles/669961.
[30] SAHA P,BELTRE A,UMINSKI P,et al.Evaluation of Docker Containers for Scientific Workloads in the Cloud[C]//Proceedings of the Practice and Experience on Advanced Research Computing.ACM,2018:11.
[31] BAUER M.The Singularity runtime [EB/OL].(2019-06-20) [2019-12-20].http://qnib.org/data/hpcw19/1_RUN_3_singularity.pdf.
[32] KINCL J,ADAMSON R.Allowing Users to Run Services at the OLCF with Kubernetes[EB/OL].(2018-05-16) [2019-08-21].https://www.fbcinc.com/e/nlit/presentations/Kincl_Adamson-Allowing_Users_to_Run_Services_at_th.pdf.
[33] Kubernetes.Kubernetes Documentation[EB/OL].(2019-02-22) [2019-08-21].https://kubernetes.io/docs/home.
[34] GAO Q.Random Talk on Container,Container Cluster Management Platform and Kubernetes[EB/OL].(2018-11-12) [2019-08-21].https://www.kubernetes.org.cn/4786.html.
[35] CHEN G.OpenShift:Enterprise Container Platform Based onKubernetes[M].Beijing:China Machine Press,2017:3-9.
[36] HINES J.OLCF Testing New Platform for Scientific Workflows[EB/OL].(2017-06-06) [2019-08-21].https://www.olcf.ornl.gov/2017/06/05/olcf-testing-new-platform-for-scientific-work-flows/.
[37] SHAH J,DUBARIA D.Building Modern Clouds:Using Docker,Kubernetes & Google Cloud Platform[C]//2019 IEEE 9th Annual Computing and Communication Workshop and Confe-rence(CCWC).IEEE,2019:0184-0189.
[38] Reeko.Solutions of Scientific Computing in the Cloud[EB/OL].(2017-12-22) [2019-08-27].http://www.reeko.net.cn/page108?article_id=245.
[39] FELDMAN M.Cloud Computing in HPC Surges[EB/OL].(2018-06-29) [2019-08-27].https://www.top500.org/news/cloud-computing-in-hpc-surges/.
[40] Kubernetes.Jobs-Run to Completion [EB/OL].(2019-12-02) [2019-12-20].https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/.
[41] Xtalpi Inc.Products and Services [EB/OL].(2018-10-25) [2019-08-29].http://www.jingtaikeji.com/products/.
[42] LIN S K.Building Container-based Scientific Computing Platform on Cloud[EB/OL].(2018-07-16) [2019-08-30].https://cloud.tencent.com/developer/article/1157276.
[43] Amazon Web Services.XtalPi Case Study[EB/OL].(2019-05-18) [2019-08-29].https://aws.amazon.com/solutions/case-studies/xtalpi/?nc1=h_ls.
[44] THURGOOD B,LENNON R G.Cloud Computing With Kubernetes Cluster Elastic Scaling[C]//Procee-dings of the 3rd International Conference on Future Networks and Distributed Systems.ACM,2019:5.
[45] WOODS C.Running Serverless HPC Workloads on Top of Kubernetes and Jupyter Notebooks[EB/OL].(2018-12-13) [2019-09-13].https://schd.ws/hosted_files/kccna18/77/woods_serverless_hpc.pdf.
[46] HE Z P,ZHANG X D,LIU Y.Microservice Architecture for Jupyter-Based Interactive Analysis Plat-form[J].Computer Systems and Applications,2019,28(8):63-70.
[47] KAVIANI N,KALININ D,MAXIMILIEN M.Towards Serverless as Commodity:A Case of Knative[C]//Proceedings of the 5th International Workshop on Serverless Computing.ACM,2019:13-18.
[48] GROβMANN M,IOANNIDIS C,LE D T.Applicability ofServerless Computing in Fog Computing Environments for IoT Scenarios[C]//Proceedings of the 12th IEEE/ACM International Conference on Utility and Cloud Computing Companion.ACM,2019:29-34.
[49] KEHRER S,BLOCHINGER W.A Survey on Cloud Migration Strategies for High Performance Computing[C]//Proceedings of the 13th Advanced Summer School on Service-Oriented Computing.IBM Research Division,2019:57-69.
[50] HONG J,DREIBHOLZ T,SCHENKEL J A,et al.An Overview of Multi-cloud Computing[C]//Workshops of the International Conference on Advanced Information Networking and Applications.Springer,Cham,2019:1055-1068.
[51] NOSSIK M,DU L,LINCOURT JR R A,et al.Security Layer for Containers in Multi-tenant Environments:U.S.Patent 10,326,744[P].2019-6-18.
[52] RATTIHALLI G,GOVINDARAJU M,LU H,et al.Exploring Potential for Non-Disruptive Vertical Auto Scaling and Resource Estimation in Kubernetes[C]//2019 IEEE 12th International Conference on Cloud Computing(CLOUD).IEEE,2019:33-40.
[1] 高诗尧, 陈燕俐, 许玉岚.
Expressive Attribute-based Searchable Encryption Scheme in Cloud Computing
计算机科学, 2022, 49(3): 313-321. https://doi.org/10.11896/jsjkx.201100214
[2] 王政, 姜春茂.
Cloud Task Scheduling Algorithm Based on Three-way Decisions
计算机科学, 2021, 48(6A): 420-426. https://doi.org/10.11896/jsjkx.201000023
[3] 刘邦邦, 易国洪, 黄祖源.
Dynamic Loading Algorithm for Docker Container
计算机科学, 2021, 48(6): 276-281. https://doi.org/10.11896/jsjkx.200500152
[4] 潘瑞杰, 王高才, 黄珩逸.
Attribute Access Control Based on Dynamic User Trust in Cloud Computing
计算机科学, 2021, 48(5): 313-319. https://doi.org/10.11896/jsjkx.200400013
[5] 陈玉平, 刘波, 林伟伟, 程慧雯.
Survey of Cloud-edge Collaboration
计算机科学, 2021, 48(3): 259-268. https://doi.org/10.11896/jsjkx.201000109
[6] 王文娟, 杜学绘, 任志宇, 单棣斌.
Reconstruction of Cloud Platform Attack Scenario Based on Causal Knowledge and Temporal- Spatial Correlation
计算机科学, 2021, 48(2): 317-323. https://doi.org/10.11896/jsjkx.191200172
[7] 蒋慧敏, 蒋哲远.
Reference Model and Development Methodology for Enterprise Cloud Service Architecture
计算机科学, 2021, 48(2): 13-22. https://doi.org/10.11896/jsjkx.200300044
[8] 张倩, 肖丽.
Review of Visualization Drawing Methods of Flow Field Based on Streamlines
计算机科学, 2021, 48(12): 1-7. https://doi.org/10.11896/jsjkx.201200108
[9] 毛瀚宇, 聂铁铮, 申德荣, 于戈, 徐石成, 何光宇.
Survey on Key Techniques and Development of Blockchain as a Service Platform
计算机科学, 2021, 48(11): 4-11. https://doi.org/10.11896/jsjkx.210500159
[10] 高雅卓, 刘亚群, 张国敏, 邢长友, 王秀磊.
Multi-stage Game Based Dynamic Deployment Mechanism of Virtualized Honeypots
计算机科学, 2021, 48(10): 294-300. https://doi.org/10.11896/jsjkx.210500071
[11] 王勤, 魏立斐, 刘纪海, 张蕾.
Private Set Intersection Protocols Among Multi-party with Cloud Server Aided
计算机科学, 2021, 48(10): 301-307. https://doi.org/10.11896/jsjkx.210300308
[12] 张恺琪, 涂志莹, 初佃辉, 李春山.
Survey on Service Resource Availability Forecast Based on Queuing Theory
计算机科学, 2021, 48(1): 26-33. https://doi.org/10.11896/jsjkx.200900211
[13] 雷阳, 姜瑛.
Anomaly Judgment of Directly Associated Nodes Under Cloud Computing Environment
计算机科学, 2021, 48(1): 295-300. https://doi.org/10.11896/jsjkx.191200186
[14] 李彦, 申德荣, 聂铁铮, 寇月.
Multi-keyword Semantic Search Scheme for Encrypted Cloud Data
计算机科学, 2020, 47(9): 318-323. https://doi.org/10.11896/jsjkx.190800139
[15] 马潇潇, 黄艳.
Publicly Traceable Accountable Ciphertext Policy Attribute Based Encryption Scheme Supporting Large Universe
计算机科学, 2020, 47(6A): 420-423. https://doi.org/10.11896/JsJkx.190700131
Full text



No Suggested Reading articles found!