计算机科学

计算机科学 ›› 2021, Vol. 48 ›› Issue (1): 319-325.doi: 10.11896/jsjkx.191100111

• 交叉与前沿 • 上一篇    下一篇

容器技术在科学计算中的应用研究

徐蕴琪1,2, 黄荷1,3, 金钟1,3   

  1. 1 中国科学院计算机网络信息中心 北京 100190
    2 中国科学院大学 北京 100049
    3 中国科学院计算科学应用研究中心 北京 100190
  • 收稿日期:2019-11-14 修回日期:2020-03-24 出版日期:2021-01-15 发布日期:2021-01-15
  • 通讯作者: 黄荷(huanghe@sccas.cn)
  • 作者简介:jocelyn_1234@163.com
  • 基金资助:
    国家重点研发计划项目(2016YFB0201700);广东省生物医药计算重点实验室(2016B030301007);中国科学院科研信息化应用工程项目 (XXH13506-202)

Application Research on Container Technology in Scientific Computing

XU Yun-qi1,2, HUANG He1,3, JIN Zhong1,3   

  1. 1 Computer Network Information Center,Chinese Academy of Sciences,Beijing 100190,China
    2 University of Chinese Academy of Sciences,Beijing 100049,China
    3 Center of Scientific Computing Applications & Research,Chinese Academy of Sciences,Beijing 100190,China
  • Received:2019-11-14 Revised:2020-03-24 Online:2021-01-15 Published:2021-01-15
  • About author:XU Yun-qi,born in 1994,postgraduate.Her main research interests include scientific computing and application service development.
    HUANG He,born in 1982,Ph.D,associate professor.Her main research interests include scientific computing and data ser-vice.
  • Supported by:
    National Key Research and Development Project(2016YFB0201700),Guangdong Provincial Key Laboratory of Biocomputing(2016B030301007) and Information Program of the Chinese Academy of Sciences(XXH13506-202).

PDF (PC)

1765

摘要： 作为一种新兴的虚拟化技术,容器能够以低廉的资源开销为应用程序和服务提供隔离的运行环境,近年来在持续集成和持续部署、自动化测试、微服务等多种业务场景中获得了广泛应用。在科学计算领域,容器技术的应用也正获得越来越多的关注。借助自身的打包能力及日益壮大的生态系统,容器技术有望为科学计算领域的生产力提升提供助力。文中对容器技术在科学计算中的应用现状进行了调研分析,并根据现有的应用实例讨论了在科学计算中使用容器及相关技术的多种方式。对不同应用模式的分析研究表明,通过提升应用程序的可移植性、改善研究的可重复性、提供非传统应用部署方案、简化云资源调度管理等多种方式,容器及相关技术可以为科学计算领域带来多方面的效率提升。

关键词: 科学计算, 容器, 容器编排框架, 云计算, 作业调度系统

Abstract: Container is a new virtualization technology that has emerged in recent years.Due to its ability to provide isolated environment for running applications and services with minimal resource overhead,it quickly explodes in popularity among enterprises and has seen wide applications in a number of business scenarios such as continuous integration,continuous deployment,automated testing and micro-services.Although not as fully utilized as in industry,the packaging ability of containers also holds promise for improving productivity and code portability in the domain of scientific computing.In this paper,we discuss how the container and related technologies can be used in scientific computing by surveying existing application examples.The different application patterns represented by these examples suggest that the scientific computing community may benefit from the container technology and the ecosystem evolving around it in many different ways.

Key words: Cloud computing, Container, Container orchestration framework, Job scheduling system, Scientific computing

中图分类号: 

  • TP391
[1] Cloud Developer.The Evolution of Container Technology[EB/OL].(2018-04-17) [2019-08-02].https://www.cnblogs.com/bakari/p/8868850.html.
[2] YANG B H,DAI W J,CAO Y L.Docker Primer[M].Beijing:China Machine Press,2015:3-6.
[3] MENAGE P B.Adding Generic Process Containers to the Linux Kernel[C]//Proceedings of the Linux Symposium.2007,2:45-57.
[4] WU Z X.Advances on Virtualization Technology of CloudComputing[J].Journal of Computer Applications,2017,37(4):915-923.
[5] WU S,WANG K,JIN H.Research Situation and Prospects ofOperating System Virtualization[J].Journal of Computer Research and Development,2019,56(1):58-68.
[6] BERNSTEIN D.Containers and Cloud:From LXC to Docker to Kubernetes[J].IEEE Cloud Computing,2014,1(3):81-84.
[7] HINDMAN B,KONWINSKI A,ZAHARIA M,et al.Mesos:APlatform for Fine-Grained Resource Sharing in the Data Center[C]//NSDI.2011,11(2011):22.
[8] Docker Inc.Docker Swarm Overview[EB/OL].(2018-02-24) [2019-08-06].https://docs.docker.com/swarm/overview/.
[9] LOU C.An Analysis of the Application Scenari-os of Docker Container[EB/OL].(2018-09-11) [2019-08-06].https://sq.163yun.com/blog/article/197408114326077440.
[10] ZHU S P.A Brief Report on Scientific Computing[J].Physics,2009(8):545-551.
[11] ZHANG X L,ZHONG Y P.A Performance Evaluation andCompare for Cluster-based Resource Management Systems[J].Application Research of Computers,2003(9):56-59.
[12] ZHOU S,ZHENG X,WANG J,et al.Utopia:A Load Sharing Facility for Large,Heterogeneous Distributed Computer Systems[J].Software:Practice and Experience,1993,23(12):1305-1336.
[13] YOO A B,JETTE M A,GRONDONA M.Slurm:Simple Linux Utility for Resource Management[C]//Workshop on Job Sche-duling Strategies for Parallel Processing.Springer,Berlin,Heidelberg,2003:44-60.
[14] NITZBERG B,SCHOPF J M,JONES J P.PBS Pro:Grid Computing and Scheduling Attributes[M]//Grid Resource Management.Springer,Boston,MA,2004:183-190.
[15] LEHTO O P.Containers,Meet HPC[EB/OL].(2015-12-30)[2019-08-12].https://medium.com/@ople/containers-meet-hpc-2aab7aa2d54a.
[16] TELFER S.The State of HPC Containers[EB/OL].(2018-08-16) [2019-08-12].https://www.stackhpc.com/the-state-of-hpc-containers.html.
[17] WANG J,HU W,ZHANG Y H,et al.Trusted Container Based on Docker[J].Journal of Wuhan University(Natural Science Edition),2017,63(2):102-108.
[18] MEDRANO-JAIMES F,LOZANO-RIZK J E,CASTAÑEDA-AVILA S,et al.Use of Containers for High-Performance Computing[C]//International Conference on Supercomputing in Mexico.Springer,Cham,2018:24-32.
[19] YOUNGE A J,PEDRETTI K,GRANT R E,et al.A Tale of Two Systems:Using Containers to Deploy HPC Applications on Supercomputers And Clouds[C]//2017 IEEE International Conference on Cloud Computing Technology and Science(CloudCom).IEEE,2017:74-81.
[20] GERHARDT L,BHIMJI W,FASEL M,et al.Shifter:Contai-ners for HPC[C]//J.Phys.Conf.Ser.2017,898:082021.
[21] JACOBSEN D M,CANON R S.Contain This,Unleashing Doc-ker for HPC[C]//Proceedings of the Cray User Group,2015:33-49.
[22] NERSC.How to Use Shifter[EB/OL].(2019-03-22) [2019-08-18].https://docs.nersc.gov/programming/shifter/how-to-use.
[23] PAN X.Shifter:Containers for HPC [EB/OL].(2017-08-23) [2019-08-18].https://www.ibm.com/developerworks/cn/linux/l-panxun-shifter-compute-container/index.html.
[24] KURTZER G M,SOCHAT V,BAUER M W.Singularity:Scientific Containers for Mobility of Compute[J].PLoS One,2017,12(5):e0177459.
[25] IDG Communications.Utilizing Containers for HPC and DeepLearning Workloads[EB/OL].(2018-05-17) [2019-08-19].https://www.cio.com/article/3269351/utilizing-containers-for-hpc-and-deep-learning-workloads.html.
[26] Singularity.Singularity Docs-Admin Guide[EB/OL].(2018-06-21) [2019-08-19].https://www.sylabs.io/guides/2.5/admin-guide/installation_environments.html.
[27] PRIEDHORSKY R,RANDLES T.Charliecloud:UnprivilegedContainers for User-defined Software Stacks in HPC[C]//Proceedings of the International Conference for High Performance Computing,Networking,Storage and Analysis.ACM,2017:36.
[28] DANDAPANTHULA N.Singularity Containers for HPC &Deep Learning[EB/OL].(2018-03-19) [2019-08-20].https://www.nextplatform.com/2018/03/19/singularity-containers-for-hpc-deep-learning.
[29] BAO X.Alibaba Cloud Supercomputing:Singularity for High-Performance Container Solutions[EB/OL].(2018-11-16) [2019-12-20].https://yq.aliyun.com/articles/669961.
[30] SAHA P,BELTRE A,UMINSKI P,et al.Evaluation of Docker Containers for Scientific Workloads in the Cloud[C]//Proceedings of the Practice and Experience on Advanced Research Computing.ACM,2018:11.
[31] BAUER M.The Singularity runtime [EB/OL].(2019-06-20) [2019-12-20].http://qnib.org/data/hpcw19/1_RUN_3_singularity.pdf.
[32] KINCL J,ADAMSON R.Allowing Users to Run Services at the OLCF with Kubernetes[EB/OL].(2018-05-16) [2019-08-21].https://www.fbcinc.com/e/nlit/presentations/Kincl_Adamson-Allowing_Users_to_Run_Services_at_th.pdf.
[33] Kubernetes.Kubernetes Documentation[EB/OL].(2019-02-22) [2019-08-21].https://kubernetes.io/docs/home.
[34] GAO Q.Random Talk on Container,Container Cluster Management Platform and Kubernetes[EB/OL].(2018-11-12) [2019-08-21].https://www.kubernetes.org.cn/4786.html.
[35] CHEN G.OpenShift:Enterprise Container Platform Based onKubernetes[M].Beijing:China Machine Press,2017:3-9.
[36] HINES J.OLCF Testing New Platform for Scientific Workflows[EB/OL].(2017-06-06) [2019-08-21].https://www.olcf.ornl.gov/2017/06/05/olcf-testing-new-platform-for-scientific-work-flows/.
[37] SHAH J,DUBARIA D.Building Modern Clouds:Using Docker,Kubernetes & Google Cloud Platform[C]//2019 IEEE 9th Annual Computing and Communication Workshop and Confe-rence(CCWC).IEEE,2019:0184-0189.
[38] Reeko.Solutions of Scientific Computing in the Cloud[EB/OL].(2017-12-22) [2019-08-27].http://www.reeko.net.cn/page108?article_id=245.
[39] FELDMAN M.Cloud Computing in HPC Surges[EB/OL].(2018-06-29) [2019-08-27].https://www.top500.org/news/cloud-computing-in-hpc-surges/.
[40] Kubernetes.Jobs-Run to Completion [EB/OL].(2019-12-02) [2019-12-20].https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/.
[41] Xtalpi Inc.Products and Services [EB/OL].(2018-10-25) [2019-08-29].http://www.jingtaikeji.com/products/.
[42] LIN S K.Building Container-based Scientific Computing Platform on Cloud[EB/OL].(2018-07-16) [2019-08-30].https://cloud.tencent.com/developer/article/1157276.
[43] Amazon Web Services.XtalPi Case Study[EB/OL].(2019-05-18) [2019-08-29].https://aws.amazon.com/solutions/case-studies/xtalpi/?nc1=h_ls.
[44] THURGOOD B,LENNON R G.Cloud Computing With Kubernetes Cluster Elastic Scaling[C]//Procee-dings of the 3rd International Conference on Future Networks and Distributed Systems.ACM,2019:5.
[45] WOODS C.Running Serverless HPC Workloads on Top of Kubernetes and Jupyter Notebooks[EB/OL].(2018-12-13) [2019-09-13].https://schd.ws/hosted_files/kccna18/77/woods_serverless_hpc.pdf.
[46] HE Z P,ZHANG X D,LIU Y.Microservice Architecture for Jupyter-Based Interactive Analysis Plat-form[J].Computer Systems and Applications,2019,28(8):63-70.
[47] KAVIANI N,KALININ D,MAXIMILIEN M.Towards Serverless as Commodity:A Case of Knative[C]//Proceedings of the 5th International Workshop on Serverless Computing.ACM,2019:13-18.
[48] GROβMANN M,IOANNIDIS C,LE D T.Applicability ofServerless Computing in Fog Computing Environments for IoT Scenarios[C]//Proceedings of the 12th IEEE/ACM International Conference on Utility and Cloud Computing Companion.ACM,2019:29-34.
[49] KEHRER S,BLOCHINGER W.A Survey on Cloud Migration Strategies for High Performance Computing[C]//Proceedings of the 13th Advanced Summer School on Service-Oriented Computing.IBM Research Division,2019:57-69.
[50] HONG J,DREIBHOLZ T,SCHENKEL J A,et al.An Overview of Multi-cloud Computing[C]//Workshops of the International Conference on Advanced Information Networking and Applications.Springer,Cham,2019:1055-1068.
[51] NOSSIK M,DU L,LINCOURT JR R A,et al.Security Layer for Containers in Multi-tenant Environments:U.S.Patent 10,326,744[P].2019-6-18.
[52] RATTIHALLI G,GOVINDARAJU M,LU H,et al.Exploring Potential for Non-Disruptive Vertical Auto Scaling and Resource Estimation in Kubernetes[C]//2019 IEEE 12th International Conference on Cloud Computing(CLOUD).IEEE,2019:33-40.
[1] 高诗尧, 陈燕俐, 许玉岚.
云环境下基于属性的多关键字可搜索加密方案
Expressive Attribute-based Searchable Encryption Scheme in Cloud Computing
计算机科学, 2022, 49(3): 313-321. https://doi.org/10.11896/jsjkx.201100214
[2] 王政, 姜春茂.
一种基于三支决策的云任务调度优化算法
Cloud Task Scheduling Algorithm Based on Three-way Decisions
计算机科学, 2021, 48(6A): 420-426. https://doi.org/10.11896/jsjkx.201000023
[3] 刘邦邦, 易国洪, 黄祖源.
面向Docker容器的动态负载算法
Dynamic Loading Algorithm for Docker Container
计算机科学, 2021, 48(6): 276-281. https://doi.org/10.11896/jsjkx.200500152
[4] 潘瑞杰, 王高才, 黄珩逸.
云计算下基于动态用户信任度的属性访问控制
Attribute Access Control Based on Dynamic User Trust in Cloud Computing
计算机科学, 2021, 48(5): 313-319. https://doi.org/10.11896/jsjkx.200400013
[5] 陈玉平, 刘波, 林伟伟, 程慧雯.
云边协同综述
Survey of Cloud-edge Collaboration
计算机科学, 2021, 48(3): 259-268. https://doi.org/10.11896/jsjkx.201000109
[6] 王文娟, 杜学绘, 任志宇, 单棣斌.
基于因果知识和时空关联的云平台攻击场景重构
Reconstruction of Cloud Platform Attack Scenario Based on Causal Knowledge and Temporal- Spatial Correlation
计算机科学, 2021, 48(2): 317-323. https://doi.org/10.11896/jsjkx.191200172
[7] 蒋慧敏, 蒋哲远.
企业云服务体系结构的参考模型与开发方法
Reference Model and Development Methodology for Enterprise Cloud Service Architecture
计算机科学, 2021, 48(2): 13-22. https://doi.org/10.11896/jsjkx.200300044
[8] 张倩, 肖丽.
基于流线的流场可视化绘制方法综述
Review of Visualization Drawing Methods of Flow Field Based on Streamlines
计算机科学, 2021, 48(12): 1-7. https://doi.org/10.11896/jsjkx.201200108
[9] 毛瀚宇, 聂铁铮, 申德荣, 于戈, 徐石成, 何光宇.
区块链即服务平台关键技术及发展综述
Survey on Key Techniques and Development of Blockchain as a Service Platform
计算机科学, 2021, 48(11): 4-11. https://doi.org/10.11896/jsjkx.210500159
[10] 高雅卓, 刘亚群, 张国敏, 邢长友, 王秀磊.
基于多阶段博弈的虚拟化蜜罐动态部署机制
Multi-stage Game Based Dynamic Deployment Mechanism of Virtualized Honeypots
计算机科学, 2021, 48(10): 294-300. https://doi.org/10.11896/jsjkx.210500071
[11] 王勤, 魏立斐, 刘纪海, 张蕾.
基于云服务器辅助的多方隐私交集计算协议
Private Set Intersection Protocols Among Multi-party with Cloud Server Aided
计算机科学, 2021, 48(10): 301-307. https://doi.org/10.11896/jsjkx.210300308
[12] 张恺琪, 涂志莹, 初佃辉, 李春山.
基于排队论的服务资源可用性相关研究综述
Survey on Service Resource Availability Forecast Based on Queuing Theory
计算机科学, 2021, 48(1): 26-33. https://doi.org/10.11896/jsjkx.200900211
[13] 雷阳, 姜瑛.
云计算环境下关联节点的异常判断
Anomaly Judgment of Directly Associated Nodes Under Cloud Computing Environment
计算机科学, 2021, 48(1): 295-300. https://doi.org/10.11896/jsjkx.191200186
[14] 李彦, 申德荣, 聂铁铮, 寇月.
面向加密云数据的多关键字语义搜索方法
Multi-keyword Semantic Search Scheme for Encrypted Cloud Data
计算机科学, 2020, 47(9): 318-323. https://doi.org/10.11896/jsjkx.190800139
[15] 马潇潇, 黄艳.
大属性可公开追踪的密文策略属性基加密方案
Publicly Traceable Accountable Ciphertext Policy Attribute Based Encryption Scheme Supporting Large Universe
计算机科学, 2020, 47(6A): 420-423. https://doi.org/10.11896/JsJkx.190700131
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
地址：重庆市渝北区洪湖西路18号    邮编：401121  
电话：023-63500828（编辑部） 023-67039612（会议/专题） 023-67039623（财务/发票）
E-mail：jsjkx12@163.com
本系统由北京玛格泰克科技发展有限公司设计开发