Generic dynamic pattrens: testing by empirical evidence

Abstract

Validation in system dynamics (SD) models can be classified into two major groups, structure and behavior validation. While behavior validation involves comparison of the model output with the real data, the structural validation focuses on the structure of the system and the mathematical relationships between the model variables and comparison with the relations in the real world. Indirect structure testing is a special type of structure testing that tests if the model generates the type of dynamic patterns that the real system is hypothesized to generate under special conditions. Indirect Structure Testing Software (ISTS) is a Hidden Markov Model based pattern recognition algorithm that implements indirect structure testing. In preliminary tests, the original version of ISTS is discovered to have certain flaws in pattern detection and classification of generic dynamic patterns. In the first part of this study, the flaws and erroneous classification problems have been resolved, by modifying/extending the algorithm and its coding application. Next, trained database of the algorithm was expanded. As an ultimate goal, we aim ISTS to be capable of detecting the generic patterns in the real noisy data sets. In addition, our goal is to improve ISTS, so that it is able to detect the generic patterns in the real data as well as the human eye can. Therefore, a Turing Test procedure has been developed, based on a pattern recognition test applied to twelve expert subjects, and their responses and ISTS results were analyzed statistically. Directly linked to the nature of our Turing test procedure, since the responses of the subjects and ISTS are qualitative, relative statistical analysis was used. First of all, it was observed that with respect to their first choices, both within subject differences and between subjects and ISTS differences were significant. On the other hand, looking at two types of outlier analysis, we obtained the results that outlier numbers of ISTS in the test are less than the average of the subjects. In our modal analysis, although the first choice results between ISTS and the subjects are not satisfactory, the results for the first or second choice analysis were satisfactory and ISTS performed as well as the subjects. In the last analysis, the new benchmark for the responses of ISTS and the subjects are our own classifications or hypotheses. Based on the results from this last test, we conclude that ISTS performs again as well as the subjects.