Comparing the Performance of Stochastic Simulators Using a Synthetic Benchmark

Herrmann, Leon (2023) Comparing the Performance of Stochastic Simulators Using a Synthetic Benchmark. Masters thesis, Institute for Visual and Analytic Computing, University of Rostock.

Text Masterarbeit-Leon_Herrmann.pdf Restricted to Registered users only Download (1MB)

Abstract

Simulating stochastic biochemical models using an exact simulation method is computationally expensive. Different implementations of multiple simulation algorithms exist, which show different run time performances for various biological models. There is currently no standardized way to compare the suitability of simulators for a collection of models. This work investigates the approach of using a parameterized synthetic model as a point of reference for run time performance comparisons. The developed synthetic model has five parameters, with the goal of reproducing the performance behaviours of biological reference models. To compare the performances between a total of 15 simulators on different models, an extensive benchmark suite was implemented in addition to the synthetic model. The results for fitting the synthetic model to a collection of reference models show, that 80% of the best fitting reference models can be represented by the synthetic model with an average relative error of 15.5%. This error decreases to a value of about 2% for the model with the best fit. Additionally, it was found that no simulator is best suited for all models. Instead, the simulation tools perform differently, depending on the parameter values used for generating the synthetic model.