Computer Science ›› 2021, Vol. 48 ›› Issue (1): 319-325.doi: 10.11896/jsjkx.191100111

• Interdiscipline & Frontier • Previous Articles     Next Articles

Application Research on Container Technology in Scientific Computing

XU Yun-qi1,2, HUANG He1,3, JIN Zhong1,3   

  1. 1 Computer Network Information Center,Chinese Academy of Sciences,Beijing 100190,China
    2 University of Chinese Academy of Sciences,Beijing 100049,China
    3 Center of Scientific Computing Applications & Research,Chinese Academy of Sciences,Beijing 100190,China
  • Received:2019-11-14 Revised:2020-03-24 Online:2021-01-15 Published:2021-01-15
  • About author:XU Yun-qi,born in 1994,postgraduate.Her main research interests include scientific computing and application service development.
    HUANG He,born in 1982,Ph.D,associate professor.Her main research interests include scientific computing and data ser-vice.
  • Supported by:
    National Key Research and Development Project(2016YFB0201700),Guangdong Provincial Key Laboratory of Biocomputing(2016B030301007) and Information Program of the Chinese Academy of Sciences(XXH13506-202).

Abstract: Container is a new virtualization technology that has emerged in recent years.Due to its ability to provide isolated environment for running applications and services with minimal resource overhead,it quickly explodes in popularity among enterprises and has seen wide applications in a number of business scenarios such as continuous integration,continuous deployment,automated testing and micro-services.Although not as fully utilized as in industry,the packaging ability of containers also holds promise for improving productivity and code portability in the domain of scientific computing.In this paper,we discuss how the container and related technologies can be used in scientific computing by surveying existing application examples.The different application patterns represented by these examples suggest that the scientific computing community may benefit from the container technology and the ecosystem evolving around it in many different ways.

Key words: Cloud computing, Container, Container orchestration framework, Job scheduling system, Scientific computing

CLC Number: 

  • TP391
[1] Cloud Developer.The Evolution of Container Technology[EB/OL].(2018-04-17) [2019-08-02].https://www.cnblogs.com/bakari/p/8868850.html.
[2] YANG B H,DAI W J,CAO Y L.Docker Primer[M].Beijing:China Machine Press,2015:3-6.
[3] MENAGE P B.Adding Generic Process Containers to the Linux Kernel[C]//Proceedings of the Linux Symposium.2007,2:45-57.
[4] WU Z X.Advances on Virtualization Technology of CloudComputing[J].Journal of Computer Applications,2017,37(4):915-923.
[5] WU S,WANG K,JIN H.Research Situation and Prospects ofOperating System Virtualization[J].Journal of Computer Research and Development,2019,56(1):58-68.
[6] BERNSTEIN D.Containers and Cloud:From LXC to Docker to Kubernetes[J].IEEE Cloud Computing,2014,1(3):81-84.
[7] HINDMAN B,KONWINSKI A,ZAHARIA M,et al.Mesos:APlatform for Fine-Grained Resource Sharing in the Data Center[C]//NSDI.2011,11(2011):22.
[8] Docker Inc.Docker Swarm Overview[EB/OL].(2018-02-24) [2019-08-06].https://docs.docker.com/swarm/overview/.
[9] LOU C.An Analysis of the Application Scenari-os of Docker Container[EB/OL].(2018-09-11) [2019-08-06].https://sq.163yun.com/blog/article/197408114326077440.
[10] ZHU S P.A Brief Report on Scientific Computing[J].Physics,2009(8):545-551.
[11] ZHANG X L,ZHONG Y P.A Performance Evaluation andCompare for Cluster-based Resource Management Systems[J].Application Research of Computers,2003(9):56-59.
[12] ZHOU S,ZHENG X,WANG J,et al.Utopia:A Load Sharing Facility for Large,Heterogeneous Distributed Computer Systems[J].Software:Practice and Experience,1993,23(12):1305-1336.
[13] YOO A B,JETTE M A,GRONDONA M.Slurm:Simple Linux Utility for Resource Management[C]//Workshop on Job Sche-duling Strategies for Parallel Processing.Springer,Berlin,Heidelberg,2003:44-60.
[14] NITZBERG B,SCHOPF J M,JONES J P.PBS Pro:Grid Computing and Scheduling Attributes[M]//Grid Resource Management.Springer,Boston,MA,2004:183-190.
[15] LEHTO O P.Containers,Meet HPC[EB/OL].(2015-12-30)[2019-08-12].https://medium.com/@ople/containers-meet-hpc-2aab7aa2d54a.
[16] TELFER S.The State of HPC Containers[EB/OL].(2018-08-16) [2019-08-12].https://www.stackhpc.com/the-state-of-hpc-containers.html.
[17] WANG J,HU W,ZHANG Y H,et al.Trusted Container Based on Docker[J].Journal of Wuhan University(Natural Science Edition),2017,63(2):102-108.
[18] MEDRANO-JAIMES F,LOZANO-RIZK J E,CASTAÑEDA-AVILA S,et al.Use of Containers for High-Performance Computing[C]//International Conference on Supercomputing in Mexico.Springer,Cham,2018:24-32.
[19] YOUNGE A J,PEDRETTI K,GRANT R E,et al.A Tale of Two Systems:Using Containers to Deploy HPC Applications on Supercomputers And Clouds[C]//2017 IEEE International Conference on Cloud Computing Technology and Science(CloudCom).IEEE,2017:74-81.
[20] GERHARDT L,BHIMJI W,FASEL M,et al.Shifter:Contai-ners for HPC[C]//J.Phys.Conf.Ser.2017,898:082021.
[21] JACOBSEN D M,CANON R S.Contain This,Unleashing Doc-ker for HPC[C]//Proceedings of the Cray User Group,2015:33-49.
[22] NERSC.How to Use Shifter[EB/OL].(2019-03-22) [2019-08-18].https://docs.nersc.gov/programming/shifter/how-to-use.
[23] PAN X.Shifter:Containers for HPC [EB/OL].(2017-08-23) [2019-08-18].https://www.ibm.com/developerworks/cn/linux/l-panxun-shifter-compute-container/index.html.
[24] KURTZER G M,SOCHAT V,BAUER M W.Singularity:Scientific Containers for Mobility of Compute[J].PLoS One,2017,12(5):e0177459.
[25] IDG Communications.Utilizing Containers for HPC and DeepLearning Workloads[EB/OL].(2018-05-17) [2019-08-19].https://www.cio.com/article/3269351/utilizing-containers-for-hpc-and-deep-learning-workloads.html.
[26] Singularity.Singularity Docs-Admin Guide[EB/OL].(2018-06-21) [2019-08-19].https://www.sylabs.io/guides/2.5/admin-guide/installation_environments.html.
[27] PRIEDHORSKY R,RANDLES T.Charliecloud:UnprivilegedContainers for User-defined Software Stacks in HPC[C]//Proceedings of the International Conference for High Performance Computing,Networking,Storage and Analysis.ACM,2017:36.
[28] DANDAPANTHULA N.Singularity Containers for HPC &Deep Learning[EB/OL].(2018-03-19) [2019-08-20].https://www.nextplatform.com/2018/03/19/singularity-containers-for-hpc-deep-learning.
[29] BAO X.Alibaba Cloud Supercomputing:Singularity for High-Performance Container Solutions[EB/OL].(2018-11-16) [2019-12-20].https://yq.aliyun.com/articles/669961.
[30] SAHA P,BELTRE A,UMINSKI P,et al.Evaluation of Docker Containers for Scientific Workloads in the Cloud[C]//Proceedings of the Practice and Experience on Advanced Research Computing.ACM,2018:11.
[31] BAUER M.The Singularity runtime [EB/OL].(2019-06-20) [2019-12-20].http://qnib.org/data/hpcw19/1_RUN_3_singularity.pdf.
[32] KINCL J,ADAMSON R.Allowing Users to Run Services at the OLCF with Kubernetes[EB/OL].(2018-05-16) [2019-08-21].https://www.fbcinc.com/e/nlit/presentations/Kincl_Adamson-Allowing_Users_to_Run_Services_at_th.pdf.
[33] Kubernetes.Kubernetes Documentation[EB/OL].(2019-02-22) [2019-08-21].https://kubernetes.io/docs/home.
[34] GAO Q.Random Talk on Container,Container Cluster Management Platform and Kubernetes[EB/OL].(2018-11-12) [2019-08-21].https://www.kubernetes.org.cn/4786.html.
[35] CHEN G.OpenShift:Enterprise Container Platform Based onKubernetes[M].Beijing:China Machine Press,2017:3-9.
[36] HINES J.OLCF Testing New Platform for Scientific Workflows[EB/OL].(2017-06-06) [2019-08-21].https://www.olcf.ornl.gov/2017/06/05/olcf-testing-new-platform-for-scientific-work-flows/.
[37] SHAH J,DUBARIA D.Building Modern Clouds:Using Docker,Kubernetes & Google Cloud Platform[C]//2019 IEEE 9th Annual Computing and Communication Workshop and Confe-rence(CCWC).IEEE,2019:0184-0189.
[38] Reeko.Solutions of Scientific Computing in the Cloud[EB/OL].(2017-12-22) [2019-08-27].http://www.reeko.net.cn/page108?article_id=245.
[39] FELDMAN M.Cloud Computing in HPC Surges[EB/OL].(2018-06-29) [2019-08-27].https://www.top500.org/news/cloud-computing-in-hpc-surges/.
[40] Kubernetes.Jobs-Run to Completion [EB/OL].(2019-12-02) [2019-12-20].https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/.
[41] Xtalpi Inc.Products and Services [EB/OL].(2018-10-25) [2019-08-29].http://www.jingtaikeji.com/products/.
[42] LIN S K.Building Container-based Scientific Computing Platform on Cloud[EB/OL].(2018-07-16) [2019-08-30].https://cloud.tencent.com/developer/article/1157276.
[43] Amazon Web Services.XtalPi Case Study[EB/OL].(2019-05-18) [2019-08-29].https://aws.amazon.com/solutions/case-studies/xtalpi/?nc1=h_ls.
[44] THURGOOD B,LENNON R G.Cloud Computing With Kubernetes Cluster Elastic Scaling[C]//Procee-dings of the 3rd International Conference on Future Networks and Distributed Systems.ACM,2019:5.
[45] WOODS C.Running Serverless HPC Workloads on Top of Kubernetes and Jupyter Notebooks[EB/OL].(2018-12-13) [2019-09-13].https://schd.ws/hosted_files/kccna18/77/woods_serverless_hpc.pdf.
[46] HE Z P,ZHANG X D,LIU Y.Microservice Architecture for Jupyter-Based Interactive Analysis Plat-form[J].Computer Systems and Applications,2019,28(8):63-70.
[47] KAVIANI N,KALININ D,MAXIMILIEN M.Towards Serverless as Commodity:A Case of Knative[C]//Proceedings of the 5th International Workshop on Serverless Computing.ACM,2019:13-18.
[48] GROβMANN M,IOANNIDIS C,LE D T.Applicability ofServerless Computing in Fog Computing Environments for IoT Scenarios[C]//Proceedings of the 12th IEEE/ACM International Conference on Utility and Cloud Computing Companion.ACM,2019:29-34.
[49] KEHRER S,BLOCHINGER W.A Survey on Cloud Migration Strategies for High Performance Computing[C]//Proceedings of the 13th Advanced Summer School on Service-Oriented Computing.IBM Research Division,2019:57-69.
[50] HONG J,DREIBHOLZ T,SCHENKEL J A,et al.An Overview of Multi-cloud Computing[C]//Workshops of the International Conference on Advanced Information Networking and Applications.Springer,Cham,2019:1055-1068.
[51] NOSSIK M,DU L,LINCOURT JR R A,et al.Security Layer for Containers in Multi-tenant Environments:U.S.Patent 10,326,744[P].2019-6-18.
[52] RATTIHALLI G,GOVINDARAJU M,LU H,et al.Exploring Potential for Non-Disruptive Vertical Auto Scaling and Resource Estimation in Kubernetes[C]//2019 IEEE 12th International Conference on Cloud Computing(CLOUD).IEEE,2019:33-40.
[1] LENG Dian-dian, DU Peng, CHEN Jian-ting, XIANG Yang. Automated Container Terminal Oriented Travel Time Estimation of AGV [J]. Computer Science, 2022, 49(9): 208-214.
[2] GAO Shi-yao, CHEN Yan-li, XU Yu-lan. Expressive Attribute-based Searchable Encryption Scheme in Cloud Computing [J]. Computer Science, 2022, 49(3): 313-321.
[3] WANG Zheng, JIANG Chun-mao. Cloud Task Scheduling Algorithm Based on Three-way Decisions [J]. Computer Science, 2021, 48(6A): 420-426.
[4] LIU Bang-bang, YI Guo-hong, HUANG Zu-yuan. Dynamic Loading Algorithm for Docker Container [J]. Computer Science, 2021, 48(6): 276-281.
[5] PAN Rui-jie, WANG Gao-cai, HUANG Heng-yi. Attribute Access Control Based on Dynamic User Trust in Cloud Computing [J]. Computer Science, 2021, 48(5): 313-319.
[6] CHEN Yu-ping, LIU Bo, LIN Wei-wei, CHENG Hui-wen. Survey of Cloud-edge Collaboration [J]. Computer Science, 2021, 48(3): 259-268.
[7] WANG Wen-juan, DU Xue-hui, REN Zhi-yu, SHAN Di-bin. Reconstruction of Cloud Platform Attack Scenario Based on Causal Knowledge and Temporal- Spatial Correlation [J]. Computer Science, 2021, 48(2): 317-323.
[8] JIANG Hui-min, JIANG Zhe-yuan. Reference Model and Development Methodology for Enterprise Cloud Service Architecture [J]. Computer Science, 2021, 48(2): 13-22.
[9] MAO Han-yu, NIE Tie-zheng, SHEN De-rong, YU Ge, XU Shi-cheng, HE Guang-yu. Survey on Key Techniques and Development of Blockchain as a Service Platform [J]. Computer Science, 2021, 48(11): 4-11.
[10] GAO Ya-zhuo, LIU Ya-qun, ZHANG Guo-min, XING Chang-you, WANG Xiu-lei. Multi-stage Game Based Dynamic Deployment Mechanism of Virtualized Honeypots [J]. Computer Science, 2021, 48(10): 294-300.
[11] WANG Qin, WEI Li-fei, LIU Ji-hai, ZHANG Lei. Private Set Intersection Protocols Among Multi-party with Cloud Server Aided [J]. Computer Science, 2021, 48(10): 301-307.
[12] LEI Yang, JIANG Ying. Anomaly Judgment of Directly Associated Nodes Under Cloud Computing Environment [J]. Computer Science, 2021, 48(1): 295-300.
[13] ZHNAG Kai-qi, TU Zhi-ying, CHU Dian-hui, LI Chun-shan. Survey on Service Resource Availability Forecast Based on Queuing Theory [J]. Computer Science, 2021, 48(1): 26-33.
[14] LI Yan, SHEN De-rong, NIE Tie-zheng, KOU Yue. Multi-keyword Semantic Search Scheme for Encrypted Cloud Data [J]. Computer Science, 2020, 47(9): 318-323.
[15] MA Xiao-xiao and HUANG Yan. Publicly Traceable Accountable Ciphertext Policy Attribute Based Encryption Scheme Supporting Large Universe [J]. Computer Science, 2020, 47(6A): 420-423.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!