Computer Science ›› 2019, Vol. 46 ›› Issue (6A): 23-29.

• Review • Previous Articles     Next Articles

Overview and Difficulties Analysis on Credibility Assessment of Simulation Models

YANG Xiao-jun, XU Zhong-fu, ZHANG Xing, SUN Dan-hui   

  1. Luoyang Electronic Equipment Test Center of China,Luoyang,Henan 471003,China
  • Online:2019-06-14 Published:2019-07-02

Abstract: Along with the developing of computing and modeling and simulation techniques,simulation models have been applied in military,social and economic areas widely in these years.Meanwhile,the capabilities of simulation mo-dels become more and more strong,and simulation systems become more and more complex.Thus,the credibility assessment of simulation models is confronted with many challenges and becomes the crucial difficult problem nowadays.This paper reviewed principal works,especially current researches on credibility assessment of simulation models.Firstly,the concept of credibility and its relationship with Verification,Validation and Accreditation(VV&A) were introduced.Secondly,the research history,paradigm and life cycle of credibility assessment of simulation models were summarized,and important methods and techniques were analyzed and classified.Finally,eight challenges of credibility assessment of simulation models were presented.The overview and difficulties analysis provide significant references for constructing the framework and innovating theories and methods for the credibility assessment of simulation models.

Key words: Credibility assessment, Difficulties analysis, Overview, Simulation models, VV&A

CLC Number: 

  • TP391.9
[13]CONWAY R W,JOHNSON B M,MAXWELL W L.Some Problems of Digital Systems Simulation [J].Management Scie-nce,1959,6(1):92-110.
[14]BIGGS A G,CAWTHORNE A R.Bloodhood Missile Evaluation [J].Journal of the Royal Aeronautical Society,1962,66(621):571-587.
[15]NAYLOR THOMAS H,FINGER J M.Verification of Compu-ter Simulation Models [J].Management Science,1967,14(2):B92-B101.
[16]MCKENNY J L.Critique of:Verification of Computer Simulation Models[J].Management Science,1967,14(2):B-102-B-103.
[17]SCHRANK W E,HOLT CC.Critique of:Verification of Computer Simulation Models[J].Management Science,1967,14(2):B-104-B-106.
[18]FISHMAN G S,KIVIAT P J.The Statisties of Diserete Event Simulation[J].Simulation,1968,10:185-195.
[19]VAN HORN R L.Validation of Simulation Results [J].Management Science,1971,17(5):247-258.
[20]SHANNON R E.Systems Simulation:The Art and Science [M].Prentice-Hall,1975.
[21]KHEIR N A,HOLMES W M.On Validating Simulation Mo-dels of Missile Systems [J].Simulation,1978,30(4):117-128.
[23]SARGENT R G.Verification and Validation of Simulation Mo-dels [J].Journal of Simulation,2013,7:12-24.
[24]BALCI O.Principles of Simulation Model Validation,Verification and Testing [J].Transaction of the Society for Computer Simulation International,1997,14(1):3-12.
[25]BALCI O.Golden Rules of Verification,Validation,Testing,and Certification of Modeling and Simulation Applications [J].SCS M&S Magazine,2010,1-7.
[26]BALCI O.Verification,Validation,and Certification of Modeling and Simulation Applications [C]∥Proceedings of the 2003 Winter Simulation Conference,2003,150-158.
[27]BALCI O.A Life Cycle for Modeling and Simulation [J].Simulation,2012,88(7):870-883.
[28]SARGENT R G,GOLDSMAN D M,YAACOUB T.A Tutorial on the Operational Validation of Simulation Models [C]∥Proceedings of the 2016 Winter Simulation Conference,2016,163-177.
[29]BALCI O.Verification,Validation and Accreditation of Simulation Models [C]∥Proceedings of the 1997 Winter Simulation Conference.1997:135-141.
[30]AHN J,DE WECK O L,STEELE M.Credibility Assessment of Models and Simulations Based on NASA’s Models and Simulation Standard Using the Delphi Method [J].Systems Enginee-ring,2014,17(2):237-248.
[31] JEBEILE J,BARBEROUSSE A.Empirical Agreement in Model Validation [J].Studies in History and Philosophy of Science,2016,56:168-174.
[32]DAMBORG M J.An Example of Error Analysis in Dynamic Model Validation [J].Simulation,1985,44(6):301-305.
[33]BARLAS Y.An Autocorrelation Function Test for Output Validation [J].Simulation,1990,55(1):7-16.
[37]LEGATES D R,MCCABE J G J.Evaluating the Use of Goodness-of-fit Measures in Hydrologic and Hydroclimatic Model Validation [J].Water Resources Research,1999,35(1):233-241.
[38]FISHMAN G S,KIVIAT P J.The Analysis of Simulation-Generated Time Series [J].Management Science,1967,13(7):525-557.
[39]MONTGOMERY D C,CONARD R G.Comparison of Simulation and Flight-test Data for Missile Systems [J].Simulation,1980,34(2):63-72.
[40]WANG X Y,MUEEN A,DING H,et al.Experimental Compa-rison of Representation Methods and Distance Measures for Time Series Data [J].Data Min Knowl Disc,2013,26:275-309.
[43]JIANG X M,MAHADEVAN S.Wavelet Spectrum Analysis Approach to Model Validation of Dynamic Systems [J].Mechanical Systems and Signal Processing,2011,25:575-590.
[45]HAUDUC H,NEUMANN M B,MUSCHALLA D,et al.Efficiency Criteria for Environmental Model Quality Assessment:A Review and Its Application to Wastewater Treatment [J].Environmental Modelling & Software,2015,68:196-204.
[46]CROCHEMORE L,PERRIN C,ANDRÉASSIAN V,et al. Comparing Expert Judgement and Numerical Criteria for Hydrograph Evaluation [J].Hydrological Sciences Journal,2015,60(3):402-423.
[47]FACHADA N,LOPES V V,MARTINS R C,et al.Model-independent Comparison of Simulation Output [J].Simulation Mo-delling Practice and Theory,2017,72:131-149.
[48]LI L Y,LU Z Z.A New Method for Model Validation with Multivariate Output [J].Reliability Engineering and System Safety,2018,169:579-592.
[49]ZHAO L F,LU Z Z,YUN W Y,et al.Validation Metric Based on Mahalanobis Distance for Models with Multiple Correlated Responses [J].Reliability Engineering and System Safety,2017,159:80-89.
[50]LIAO W C,ZHANG J,ZHENG X P,et al.A Generalized Validation Procedure for Pedestrian Models [J].Simulation Modelling Practice and Theory,2017,77:20-31.
[51]OLSEN M M,RAUNAK M,SETTEDUCATI M.Enabling Quantified Validation for Model Credibility [C]∥Proceedings of the 50th Computer Simulation Conference.2018:1-10.
[52]KWAG S,GUPTA A,DINH N.Probabilistic Risk Assessment Based Model Validation Method Using Bayesian Network [J].Reliability Engineering and System Safety,2018,169:380-393.
[53]LI Y L,WANG X J,WANG C,et al.Non-probabilistic Bayesian Update Method for Model Validation [J].Applied Mathematical Modelling,2018,58:388-403.
[54]LAMPERTI F.An Information Theoretic Criterion for Empirical Validation of Simulation Models [J].Econometrics and Statistics,2018,5:83-106.
[55]ATKINSON A D,HILL R R,PIGNATIELLO J J J,et al. Wavelet ANOVA Approach to Model Validation [J].Simulation Modelling Practice and Theory,2017,78:18-27.
[56]ATKINSON A D,HILL R R,PIGNATIELLO J J J,et al. Wavelet ANOVA Bisection Method for Identifying Simulation Model Bias [J].Simulation Modelling Practice and Theory,2018,80:66-74.
[57]SONG H F,LIU J Y,SCHNIEDER E.Validation,Verification and Evaluation of a Train to Train Distance Measurement System by Means of Colored Petri Nets [J].Reliability Engineering and System Safety,2017,164:10-23.
[58]SONG H F,SCHNIEDER E.Development and Validation of a Distance Measurement System in Metro Lines [C]∥IEEE Transactions on Intelligent Transportation Systems.2018:1-16.
[59]GUARRO S B,YAU M K,OZGUNER U,et al.Formal Framework and Models for Validation and Verification of Software-Intensive Aerospace Systems [C]∥Proceedings of AIAA SciTech Forum.2017:1-10.
[65]王石,伍丁红,戴金海.基于 Bayes 网的武器装备仿真可信度评估[J].系统仿真学报,2011,23(6):1082-1085,1103.
[66]BALCI O,ADAMS R J,MYERS D S,et al.A collaborative eva-luation environment for credibility assessment of modeling and simulation applications [C]∥Proceedings of the 2002 Winter Simulation Conference.214-220.
[70]YANG Y N,KUMARASWAMY M M,PAM H J,et al.Inte-grated Qualitative and Quantitative Methodology to Assess Validity and Credibility of Models for Bridge Maintenance Management System Development [J].Journal of Management in Engineering,2011,27(3):149-158.
[71]PADILLA J J,DIALLO S Y,LYNCH C J,et al.Observations on the Practice and Profession of Modeling and Simulation:A Survey Approach [J].Simulation,2018,94(6):493-506.
[72]SARGENT R G,BALCI O.History of Verification and Validation of Simulation Models[C]∥Proceedings of the 2017 Winter Simulation Conference.2017:292-307.
[1] WANG Yu-fei, CHEN Wen. Tri-training Algorithm Based on DECORATE Ensemble Learning and Credibility Assessment [J]. Computer Science, 2022, 49(6): 127-133.
[2] HU Yu-jia, GAN Wei, ZHU Min. Enhancer-Promoter Interaction Prediction Based on Multi-feature Fusion [J]. Computer Science, 2020, 47(5): 64-71.
[3] YI Wen-ying and LI Bo. Research on Verification and Validation of Agent-based Simulation Models [J]. Computer Science, 2015, 42(Z11): 460-463.
[4] GUAN Hua,YING Shi,JIA Xiang-yang,JiangCaoQing and WANG Yi-bing. Overview on Exception Handling in Service Oriented Software [J]. Computer Science, 2013, 40(4): 1-8.
[5] GUO Li-juan,HU Jun,ZHANG Jian. Improved Design and Implementation of T-CBESD Based on On-the-Fly Verification Methods [J]. Computer Science, 2011, 38(10): 145-151.
[6] . [J]. Computer Science, 2008, 35(10): 25-27.
[7] . [J]. Computer Science, 2007, 34(12): 283-285.
Full text



No Suggested Reading articles found!