Verification and validation of simulation models

 

Verification and validation

Model Building:

- The real system and their interactions among various components should be analyzed.
- The domain knowledge can also be acquired from the interaction with concerned people.
- As the model development proceeds, new questions arises and the process of learning system behaviour and structure takes place again.
- Then a conceptual model is constructed with a collection of assumptions and hypotheses.
- Finally, the conceptual model is implemented using various simulation software incorporating assumptions into the worldview.
- Hence, model building is an iterative process of domain knowledge acquisition and model development.

---

Verification:

- Verification is concerned with building the model correctly.
- It is performed by comparing the conceptual model to the computerized simulation implementation.
- It tests whether the given model is implemented correctly in the simulation software.

---

Validation:

- Validation is concerned with building the correct model.
- It is performed by calibration of the model.
- Calibration is the iterative process of comparing the model to the actual system behaviour.
- It attempts to confirm that a model represents the real system accurately.

---

Verification of simulation models

Considerations to be used in Verification Process:

1. The operational model should be checked by someone other than the developer, preferably an expert in the simulation software being used.
2. Make a flow diagram that contains all logically possible action a system can perform when an event occurs.
3. Closely examine the model output for reasonableness under variety of input parameters.
4. Have the operational model print input parameters when the simulation ends to check if they are not altered.
5. Make the operational model as self-documenting as possible.
6. Verify animated operational model imitates the actual system.
7. Debugger should be used during simulation model building.
8. Graphical interfaces are recommended.

---

Calibration and validation of models

Naylor and Finger Simulation Model Validation:

Naylor and Finger proposed three step approach for validation process. These steps are as follows:
1. Face Validity
2. Validate Model Assumptions
3. Input - Output Transformation

---

Face Validity:

- A model should appear reasonable on its face to model users and to those who knows about the real system that is being simulated.
- A model should be designed with high degree of realism regarding system structure and behaviour through reliable data.
- The potential users should also be involved in the validation process to aid in identification of model deficiencies and optimizing those deficiencies to produce better model. This process is termed as structural walkthrough.
- Sensitivity analysis is also used for face validity of the model. It analyses the effect on output when there is change in input parameters.
- Sensitivity analysis is done through appropriate statistical techniques.

---

Validate Model Assumptions:

- There are many assumptions that are made during model building process.
- These assumptions are of two types: structural assumption and data assumption.
- Structural assumptions involves simplification and abstraction of reality.
- Data assumptions should be done based on collection of reliable data.
- For example - Consider a bank system in which customer are queued before providing service.
- The structural assumptions may be - Customers form a single queue which is served by multiple tellers - Customers form one line for each teller - The number of teller should be fixed or variable.
- These structural assumptions should be validated by actual observation during appropriate time periods and also by discussions with the managers and tellers.
- The data assumptions may be - interarrival times of customers during peak hour - interarrival times of customers during slack period - service time for personal accounts and so on.
- These data assumptions should be validated by consultation with bank managers. The validation is done by using goodness-of-fits tests such as chi-square test or Kolmogorov-Smirnov tests.

---

Input-Output Transformations:

- It involves validating whether the model can predict the future behaviour of the real system when the model input data match the real inputs and when a policy implemented in the model is implemented at some point in the system.
- In this validation phase, the model accepts values of the input parameters and transforms them into output measures of performance.
- The modeler uses the historical data reserved for validation purposes only.
- The main responses of interest should be used as the primary criteria for model validation.
- If in future, the model is used for different purpose, it should be revalidated in terms of new response of interest.
- For the complete input-output validation, at least one set of input conditions should be collected from the system data so as to compare to model prediction.

---

Iterative Calibration for Validation:

- Iterative calibration means to validate the model with the real system, look out for the places for betterment of the models and revising the model to form next better model repeatedly until a satisfiable model is not achieved.
- The initial model is developed and is calibrated using Naylor-Finger calibration steps with the real system. It is then revised and a first revision model is generated.
- The first revision model is then calibrated with the real system. It is revised to form a second revision model.
- This process is continued until the model becomes acceptable.

---