A Quantitative Evaluation Framework for the Benefit of Building Information Modeling for Small and Medium Enterprises Leveraging Risk Management Concepts

. Building Information Modeling plays an increasingly important role in today's construction sector due to its growing prevalence among practitioners and the increasing legal obligation to use it in public projects. The general and most obvious benefits such as improved communication/coordination or rapid generation of various planning alternatives are widely accepted. However, there is a lack of a generic framework to quantify these benefits. This is especially important from the perspective of small and medium sized enterprises because larger investments - as they may occur through software and training when implementing BIM - should be balanced against a traceable return-on-in-vestment. With a few exceptions, other research focuses on qualitative or non-monetary assessment parameters to evaluate these benefits. Such an approach may be too vague in the context of small and medium-sized enterprises to justify the necessary strategic decisions on a BIM changeover. In this study we propose a generic framework for quantitative evaluation of the BIM benefit. This evaluation is based on the BIM-impact on contingency estimation in construction risk management relying on Monte Carlo simulations. Semi-structured interviews and questionnaires with practitioners are used to evaluate how risk factors and subsequently contingencies can differ from each other in BIM-projects and Non-BIM projects.


BIM and the Assessment of its Benefits
Building Information Modeling (BIM) is a process for creating and managing information throughout the lifecycle of a building or an infrastructure [1].This process results in a Building Information Model, a digital representation of the built asset, developed collaboratively by designers, construction firms and clients.Collaboration is the key aspect of the BIM methodology since it enhances decision making, increases productivity resulting in a greater whole life value for the asset [2].For this reason, BIM plays an increasingly important role in today's AEC sector and its adoption is growing faster, year after year.E.g., in the United Kingdom, the number of practitioners using BIM has increased from 13% in 2011 to 69% in 2019 [3].This exponential trend addresses also the incremental demand for BIM from the client's side and especially from public organizations.In fact, since the introduction of BIM in the European Directive on Public Procurement (24/14/EU) in 2014, several governments across Europe are pushing the mandatory use of BIM in public works with the aim of reducing waste, delays and costs, while increasing the quality of their built environment [4].
Beyond strategic expectations from the AEC community on BIM, its use has already evidenced many benefits also at both project and company level, such as: better communication, coordination and rapid generation of design alternatives [5,6].However, even if these benefits are widely accepted, previous research in this field still consists in observations from singular case studies [5] and, as stated by [7], no unified nor global approach has been proposed so far.Since implementing BIM in organizations is a strategic investment, the lack of a framework to quantitatively measure its benefits is critical especially from the perspective of small and medium sized enterprises (SMEs) [6], where investments have to be balanced against a clear return-on-investments (ROI).
Measuring the ROI of BIM has different finalities: (i) understanding the impact on the organization performance; (ii) measuring benefits; (iii) considering the risks associated to the benefits; (iv) collecting data useful for the IT investment analysis [8].While previous research investigated the relationship between benefits and how these effects ROI, to our best knowledge there are no scientific studies that are concerned with the relationship between potential risk factors in a construction project and the mitigation improvements achievable using BIM.This interplay might affect ROI considerations when thinking of BIM implementations.

Problem Statement
As being outlined above the benefit of the BIM approach for implementing companies especially for SMEsis not reliably measurable in a direct way.One reason for this is that there is little long-term experience with the use of BIM, and it is not sufficiently widespread within the construction industry.Secondly, the BIM impact on the performance of a construction project is difficult to relate to a reference or benchmark of a traditional approach.This is due to the unique characteristics of buildings: Two identical buildings with identical boundary conditions (same subsoil conditions, neighboring buildings, weather, team of project participants, contractor and subcontractors, etc.) would have to be built in order to compare the two approaches.However, even in this case, no direct comparability would be guaranteed, as the project team would probably learn from the first realization, and the second one would therefore run better and faster in any case, which would dilute the assessment of the BIM influence.Therefore, we follow an indirect way to potentially assess the impact of BIM by considering different BIM functionalities and their impact on risk management in construction projects.
More precisely, we use methods of probabilistic risk management.In the field of probabilistic risk management there are numerous known risk factors (such as poor communication and coordination.Their influence on the monetary contingency to be made available in case of risk occurrence is sufficiently described in the literature.However, what is not described in the literature is the potential influence of BIM on these risk factors.This paper aims at contributing to fill this gap. We further hypothesize that several BIM functionalities have a positive influence on certain risk factors, both in terms of the likelihood of its occurrence and financial impact.Consequently, the contingency to be held available would be reduced compared to non-BIM projects.Using this indirect way, we attempt to describe the BIM benefit quantitatively.Accordingly, this paper investigates two aspects of research, namely (i) the quantitative evaluation of the BIM approach and (ii) the BIM influence on known risk factors in construction.

State of the Art: Risk Management in Construction
According to the PMBOK® Guide [9], risk management plays a crucial role in project management and is defined therein as "systematic process of identifying, analyzing, and responding to project risks".The extent of a risk is determined by two factors: The "likelihood of occurrence" and the "impact" of a risk.Both determinants have different probabilities.The first specifies the likelihood of the risk occurring, while the second quantifies the probable magnitude of the financial impact in case of occurrence [10].Two approaches are used here: Qualitative and quantitative risk assessment strategies.
Within the existing frameworks for qualitative risk assessment, the respective risk impact and the likelihood of occurrence are classified merely trough few levels (high, medium, low) and within this classification, percentage supplements in terms of budget are added on the estimated production costs [11].
Quantitative risk assessment on the other hand requires a more detailed consideration from an economic and financial perspective.The individual risks are assigned a potential impact range in monetary units accompanied by probability distributions as well as binomial distributed likelihoods of occurrence.A resulting quantitative measure of the overall risk impact of a project is the value at risk as a loss-oriented risk measure.This value is estimated with the aid of numerical simulation methods such as the Monte Carlo simulation (MCS) method.
The MCS method is used as a computer-supported, mathematical instrument for solving complex tasks.In general, the Monte Carlo method is understood as the use of random numbers for the solution of stochastically describable problems, even if these problems are not stochastic in their origin [12,13].
It is possible to distinguish two fields in which MCS is typically used: The first field contains deterministic problems, where also exact analytical solutions exist.MCS is used here when simulation results, as an approximate solution, can describe the actual solution with enough accuracy.The loss of accuracy that is associated with MCS is consciously accepted and part of a trade-off with simplicity and velocity within the process of finding a solution.This trade-off yields frequently to the selection for MCS, as analytical solutions, in these fields, are often very computationally and resource intensive.This use of MCS has a high significance within the natural sciences and mathematics.Application examples for deterministic problems that can be solved with MCS are the calculation of integrals and the solution of ordinary and partial differential equations [13].
The second problem area comprises stochastic problems.These are characterized by the fact that the input parameters and the resulting target variables are to a certain extent subject to chance.In stochastic problems, MCS can be used to investigate situations where either no analytical solution exists or input parameters comprise random variables [14].In the area of risk analysis, MCS can be used to aggregate individual risks to form an overall risk, multi-scenario simulations as well as for decision optimization.

Research Strategy
The research strategy followed in this paper is a combined approach of (i) elaborating a probabilistic risk model containing risk factors in accordance with [15].The calibration of the risk model is performed by analyzing literature, results from a questionnaire as well as findings of semi-structured interviews with construction experts.Secondly (ii), the research strategy consists of performing the MCS to assess the impact of BIM on contingency estimation in construction from the risk management perspective.By doing so, we aim to measure the added value of BIM in an indirect way.While the use of MCS for contingency estimation in construction is widely described in research [15][16][17], the analysis of the BIM influence within such a probabilistic system is new to our best knowledge.This evaluation is based on the framework for benefit evaluation of Target Value Design (TVD) on construction costs by [18].TVD is a method from Lean Construction management to create more value for the client.One dimension of value in Lean Construction consists in staying on budget.In the mentioned evaluation framework, a theoretical cost-breakdown model was postulated, which divides the individual cost components for construction projects into cost of work, contingency and profit.
Their findings on the statistical analysis of TVD and non-TVD projects impose a reduction of contingency in TVD projects.The authors denominate the drivers for contingency as "forces" that drive up total construction costs.These forces are identified by the authors as e.g.poor communication and coordination, lack of trust, change orders etc.They argued further that in TVD projects, these negative drivers can be reduced and hence the required contingency is likely to be reduced, too.
Drawing a line back to this research, these mentioned "forces" can be also considered risk factors in construction.The implementation of BIM might affect some of these risk factor positively.However, it remains unclear to which extent BIM affects these risk factors, and what effects this has on contingency considerations.
Therefore, the proposed framework for quantitative BIM benefit evaluation is based on a probabilistic risk model to determine the effect of BIM on contingency estimation.The underlying assumption, that BIM can actually contribute to contingency reduction through the mitigation of risk factors is shown in Fig. 1 as adaptation of TVD projects by [18].For the purpose of BIM benefit assessment, we attempt to determine a so-called BIMbased contingency delta (BIM-C-Δ), which might be caused by certain risk factors.Thus, the BIM-C-Δ does not represent the potential overall BIM benefit in that regard, since we are not going to consider all factors that might have an impact on contingency estimation, but only the top risk factors which might be affected through the use of BIM.This means, the potential overall BIM-benefit could be even higher if all risk factors were examined.
Consequently, this study aims to examine the following hypotheses: 1. Null-Hypothesis  0 : µ  ≤ 0 The implementation of BIM has no positive impact on the required contingency in cost estimation.2. Alternative-Hypothesis   : µ  > 0 The implementation of BIM has a positive impact on the required contingency in cost estimation.
For assessing these hypotheses, we elaborate a risk register that contains a total five risks factors which might be affected by BIM.The elaboration of this risk register is described in detail in section 4.1.The risks contained therein are transferred into a questionnaire that is handed out to selected local construction experts to evaluate the extent to which the likelihood of occurrence varies under BIM application.In this study, we analyze a total of 22 responses to this questionnaire.
To test the presented hypotheses, we conducted a paired t-test on the questionnaire results.The test results are discussed in section 4.2.
Presuming a statistically significant difference of risk perception in the two scenarios (rejection of  0 ), we apply MCS to simulate the needed contingency for a defined project scenario (residential construction in the order of magnitude up to 10 Mio.€ contract volume), taking into account the questionnaire's results for configuration of the probabilistic risk model in terms of likelihoods, impacts and dispersion.

4
Proposal for Quantitative BIM Benefit Evaluation

Elaborating the Risk Register
The risk register used in this study is based on the overall top 10 risks identified in a literature review of the most common risks in construction [19].These are the risks that occur most frequently according to that study.We contrast these risks with the 10 risks that are most significant according to [20].The assessment criteria for significance here are the likelihood of occurrence and the potential financial risk impact in case of occurrence.
We have made an intersection of these two lists and we have neglected all those risks which are not clearly influenced by BIM in terms of likelihood of occurrence with regard to the mentioned BIM benefits in section 1.1.We have given the five remaining risks an aggregate name and assigned them to risk categories in line with [17].Additionally, we ranked them according to their relative likelihood of occurrence and financial impact as reported by [21].The resulting list is shown in Table 1 and forms the basis for the probabilistic risk model in this study.

Expert Questionnaire & Hypotheses Testing
We elaborated a questionnaire for the configuration of the probabilistic risk model.The questionnaire goals are two-fold: (i) to evaluate how the likelihood of occurrence of the risk factors changes under BIM usage from the experts' point of view for the purpose of hypotheses testing, and (ii) to find out which dispersion parameters are appropriate for the impact distribution for MCS in the case the Null-Hypothesis has to be rejected.A total of 22 responses have been analyzed when this study was conducted.
As an example, out of these risk factors, the first risk factor (M1) is concerned with whether a project can be delayed due to decisions not taken or taken too late because the information needed was not (sufficiently) available.The experts were asked to express their opinion on the likelihood of the occurrence of this risk factor on a 4-point Likert-scale with the options (i) very likely; (ii) likely; (iii) unlikely and (iv) very unlikely in the two scenarios of traditional and BIM-based project delivery.By BIM project delivery, we mean a full use of all available functionalities (e.g.collision check, common data environment, automatic plan derivations, model-based quantity take-off etc.) by all project participants.The results are presented in Fig. 2.

Fig. 2. Risk factor M1
These results show a change in the perception of risk occurrence from very likely to unlikely.In fact, the number of votes for very likely has been halved, which in turn increased the votes of an unlikely risk event in the BIM-based case.The number of votes for likely have approximately remained the same.
Furthermore, the respondents were asked to estimate the potential financial impact of the considered risk factors.For the risk factor M1 it has been estimated in the range from significant (68%) to critical (23%) with a medium degree of statistical dispersion, meaning that the financial impact of this risk factor is subject to uncertainties (not shown in the graphic).The findings for the other four considered risk factors are of similar appearance, so that a detailed description of those is omitted at this point.Subsequently, and in accordance with [22], who studied probability related terms and their corresponding values as percentages, the different options of the Likert-scale of the questionnaire have been assigned the following percentages as a numerical likelihood of occurrence as an assumption: (i) Very likely = 65%; (ii) Likely = 50%; (iii) Unlikely = 7,5%; (iv) Very Unlikely = 2,5%.

45,50%
As a next step, we calculated the mean of the responses by creating ordinal data through assignment of numbers from 1 to 4 to the answer options (1 = Very unlikely; 2 = Unlikely; 3 = Likely; 4 = Very unlikely).The mean here is used to find out general tendency of the responses with respect to the numerical likelihood as percentages and eventually for hypotheses testing, making use of a paired t-test.The question to be analyzed with this paired t-test is whether BIM does impact the required project budget or not.Hence, the difference of the mean in both cases is in question, hypothesizing that µ d ≤ 0 (= H 0 ).
The paired t-test with an α-level of 0,1 and under t-distribution with 4 degrees of freedom has shown that the null hypothesis H 0 is to be rejected and the alternative hypothesis is to be accepted, which means that BIM has a statistically significant (positive) effect on the risk occurrence according to the questionnaire findings given the considered risk factors.It remains to be analyzed whether this discrepancy in perception on the effect on construction risks between traditional and BIM-based projects is also measurable trough numerical simulation of risk events.
The next section presents the probabilistic risk analysis framework to quantify this effect in terms of required contingency of a given project budget.In this study, the project budget is assumed to be 10 million € (project scenario: residential construction).Simulation results do not claim to be reliable or valid in terms of absolute numbers but are intended to show the relative difference between the two project delivery approaches (traditional vs. BIM-based) and to present this idea of measuring the BIM benefit in an indirect way to the scientific community.

Probabilistic Risk Model
This list of risk factors given in Table 1 is the fundamental element of the probabilistic risk model for performing the MCS for contingency estimation.Fig. 3 shows the risk model in the BIM-based configuration exemplarily.

Fig. 3. Risk model BIM-based project delivery
It is conceptually derived from the risk register presented by [23] and composed of (i) the risk factors themselves and a short description; (ii) a quantitative likelihood of occurrence as derived from the questionnaire results; (iii) a qualitative impact in the range from negligible to crisis; (iv) a rough financial order of impact in terms of euros [€] as agreed with local construction experts; (v) the type of probability distribution for that impact and form parameters for stochastic analyses (mean and standard deviation) as well as (vi) the presumed correlation of the single risk factors.
In both cases, traditional and BIM-based project delivery, we assume a lognormal distribution for the probability density function (PDF), since only mathematically positive values for the financial impact are allowed (even though risk occurrence would create a negative impact on the project budget).Moreover, this selection is in line with other research on cost estimation and risk management in construction which argues that lognormal distributions fit construction cost components generally the best [17].
Regarding the estimation of form parameters of the PDF, we abstracted the screened literature and findings of semi-structured interviews with representatives of a total of five enterprises from the South Tyrolean construction sector.These enterprises comprise SMEs from the fields of timber construction, civil engineering, shell construction and HVAC.Their representatives were asked to roughly estimate the given risk factor's impact in case of occurrence.Together with literature indications [21] as well as the questionnaire results, we configured the risk model for both cases, traditional project delivery approach and BIM-based project delivery approach.The likelihood of occurrence was modelled as binomial distribution with form parameters according to the percentage values according to [22].
In construction cost estimation, simulation-based analysis is highly affected by potential correlations of single cost factors [24].The cited author demonstrates a slight to medium positive correlation of cost components which was confirmed in expert discussions for the considered risk factors and incorporated accordingly in the probabilistic risk model.

Preliminary Simulation Results
The Monte Carlo simulation was run 10000 times in both cases.Fig. 4 shows the simulation results in a histogram.It can be seen that the maximum (2,56% discrepancy) and the standard deviation (1,12% discrepancy) differ only slightly from each other, while the mean (µ) and median (x ) show a relatively large discrepancy: In the BIMbased case, the mean is more than 20% below the traditional project delivery.The BIMbased median is even 28,21% lower than in the traditional case.In both cases the distributions can be considered as approximately symmetric.Only a slight skewness to the right in the BIM-based distribution can be noted (0,39).Given the kurtoses of 2,12 (traditional) and 2.24 (BIM-based), both resulting distributions can be considered as platykurtic and thus having relatively low and broad central peaks compared with the normal distribution.The mode corresponds in both cases to 0€.However, in the BIM-based case the value of 0€ for the required contingency occurs almost twice as often as in traditional project delivery.

Inferences from Results & Discussion
The simulation results for the selected configuration of the risk model and the resulting descriptive statistics allow the following statements: There are relatively large differences in the mean (µ) and median (x ) between traditional and BIM-based project delivery (21,52% and 28,21%, respectively), while the minima and maxima, as well as the standard deviation σ x , are relatively close to each other.The standard deviation itself appears as approx.60% respectively 76% of the mean.Within two standard deviations there are more than 95% of all simulated values, and within three standard deviations there are more than 99,7% of all simulated values which is an indication for normal distribution.A high standard deviation indicates many outliers, which means that the median is the more appropriate single value to describe the whole sample with only one value.From this background, the median difference of almost 30% indicates a significantly positive effect of the BIM approach on contingency estimation.However, these results should only be interpreted as a general tendency, since on the one hand the modeling and simulation results are associated with some limitations (see section 7) and on the other hand the results from the mere histogram analysis with the usual confidence percentile of, for example, P90, only shows relatively small differences.Since these high confidence intervals are usually the ones on the basis of which strategic decisions are made (such as investment decisions), it remains to be checked whether the proven relative differences between traditional and BIM-based project delivery are sufficient to justify such -especially for SMEs drastic -changes as a complete BIM changeover.

Limitations, Conclusion & Outlook
This study has some worth mentioning limitations: only few risk factors have been considered in the probabilistic risk model, thus, the underlying risk register should be expanded in further studies.Furthermore, the assumed financial impact is only a rough estimation by the construction experts surveyed with reference to a hypothetical project scenario.These impacts would have to be determined for a larger number of different project scenarios and in a more robust manner -for example, by evaluating historical project data -which is out of scope of this study, since the focus is on presenting the benefit evaluation concept itself.Even if this were to be done in the future, it could still be argued with the criticism that every project in the construction industry is different and it is therefore difficult to make general statements about the BIM impact.This criticism would still be justifiedas explained in the section on related work in other examples -but in our view, the Monte Carlo simulation approach helps to refute this counter-argument, because this type of numerical simulation aims to play through as many random scenarios as possible in order to derive general inferences.
With respect to the chosen probability distributions there is also uncertainty, especially since the choice of the distribution has a considerable influence on the simulation results as explained by [25].However, the presented probabilistic risk model aims at providing reasonable assumptions for the final choice of the log-normal distributions, which are explained in section 4.1.In addition, this benefit assessment framework aims at providing an example of the relative difference in contingency requirements comparing a traditional and BIM-based project delivery approach, which we believe is possible even if the absolute numbers are not sufficiently reliable.
In the light of future questionnaire extensions, this study represents an invitation to the research community to evaluate the influence of BIM on risk factors in a supraregional context in order to obtain a more complete picture of the appropriateness of the suggested indirect evaluation approach.
To conclude the results of this study, the initially stated research objectives of (i) formulation of a quantitative BIM-benefit evaluation framework and (ii) analysis of the BIM influence on known risk factors in construction have been both addressed and validated respectively through the simulation and questionnaire results.
Given the reference to TVD in this study, it may also be useful to examine projects where both BIM and TVD have been used for the combined impact on contingency determination.This could lead to a contribution to the body of knowledge in the area of practical BIM benefits, as well as new BIM-Lean synergies that have not yet been explored.
Future considerations using this BIM benefit evaluation framework should be also put in relation to the incurred costs, which require a changeover to BIM-based project delivery.In the area of BIM implementation, these costs mainly concern the investment in software, training, process changeover and personnel.Only in this overall context, quantitative BIM evaluation can provide reliable information about strategic management investment decisions of SMEs.

Table 1 .
Considered risk factors