Model Validation
Regardless of the approach you choose, validation is a key step before using results to draw conclusions. In fact, validation is a step that is generally performed repeatedly during the course of model development, as you continue to add enhancements. By verifying fundamental behaviors of your model at each step, you can maintain a high degree of confidence, and easily identify which particular changes are responsible for new, unexplained behavior. In contrast, if you make many simultaneous changes before examining the model’s behavior, you will then be unable to determine which changes are responsible for your observations, and to what extent each change has contributed. In short, after you build your first basic model of your system, you should progress incrementally, performing validation at each step.
Even though validation sounds like a formal term reminiscent of a mathematical proof, or formal verification, it need not be handled in that manner. Rather, validation is the process of maintaining confidence in your model’s equivalence to the real system and in its ability to generate useful results. Therefore, you are again responsible for determining what is necessary to achieve this confidence. Typically, you will make use of a combination of several techniques, outlined below:
-
Common Sense and Intuition – Simplistic as it may seem, this is your most important tool in model validation. Even if you cannot tell if an answer is correct, or what its degree of accuracy is, you should have an opinion on whether an answer is in the right ball park, or significantly different than what you would consider plausible. If it is not, the model could still be correct, but it’s time to investigate further.
Your common sense and intuition may come from your experience with the real system, or with the technologies at hand. You may also be relying on crude mathematical calculations you have performed, which serve as a boundary analysis. The boundary analysis is a simple calculation that you expect to definitely be lower or higher, as the case may be, than the answer your model will produce. In other words, if you know that your model accounts for certain effects that your calculation does not, try to determine if those effects would modify the result in one direction or another.
For example, will accounting for packet fragmentation and overhead increase application response times or decrease them? You would probably answer increase them, so if your simulation model does fragmentation, and your simple calculation does not, you might expect the simulation model to predict higher response times.
-
Measurement – This is the most common approach to validation and the one most people think of first. Some people refer to it as baselining against the real system. Of course, it can only be done if the real system, or some portion of it, is actually accessible. Measurement tools can consist of network analyzers, such as Network General’s Sniffer, or those manufactured by Wandel & Goltermann, Hewlett Packard, or one of many other vendors; or they can be as simple as a stopwatch for measuring the interval required to complete an activity. Some networks, applications, or protocols are also instrumented to report on certain aspects of their performance.
Here again, you should have a notion of what the differences are between your model and the real system. When the results differ, as they typically do initially, you will want to make a judgement about whether the differences can be attributed to simplifications in your model. Is your model operating under all of the same conditions as the real system?
As an important side-note, many people think of measurement purely as a validation tool. In fact, measurement can be a powerful modeling tool as well.
-
Alternative Models – By building another model using a different approach, you can gain insight into how both models behave and which one will do the best job for you. In other words, which is providing more-valid results for the questions you want to answer under the operating conditions that matter? The alternative model might not be your own. It could have already been provided or had its results published by another party. It could also be a model of a completely different nature, such as a mathematical or analytical model. Essentially, what you are doing here is very similar to the technique of performing measurements. However, in this case, your measurements are taken in another model of the system instead of the system itself.
-
The Control Experiment – This is a technique familiar from high-school science classes. Build a test case in which you know what the results should be with a high degree of confidence. Typically, this is done by using extreme operating conditions to isolate particular behaviors of the model. For example, what is the application response time if the server is infinitely fast and transactions are very small so that requests and responses can easily fit into one packet? Under these conditions, you can generally do a good job of making a simple manual estimate and compare against your simulation. A good comparison will build your confidence for moving to the next step: changing one aspect of the test case. In this example, that might be changing the transaction size. The fundamental concept of the control experiment is to remove most behaviors that are difficult to explain and to use common sense to perform validation in their absence. Then use incremental analysis, described below, to gradually re-introduce some of the other complex mechanisms in the system.
-
Incremental Analysis – Making individual changes rather than many changes at once has already been emphasized as a sound strategy. As a validation technique, incremental changes and analysis of their impact helps you understand if each change makes sense. In the final phases of your project, you will use this information to explain what were the significant factors in determining performance of your system. This will also allow you to detect when certain mechanisms act in combination to produce interesting behaviors. For example, a slow link may not be a problem for your system’s performance, nor may a low but non-zero rate of frame loss; but the combination of the two could cause your end-to-end response times to increase dramatically. In summary, the purpose of incremental analysis is to experiment with the model’s parameters to gain confidence with the behavior of individual features.
Validation is a rewarding process as long as you are receiving confirmation that your model is providing equivalence with the real system, as you are expecting. However, this will not always happen, so you must have an approach to follow when validation is negative. The following are some typical approaches you can take in such a situation:
- Return to the previous validated state – If you have performed a validation of the model at a previous stage, save your current model and return to its previous form (this highlights the value of saving each important version of your model, particularly at each validation). First, make sure that you can recreate your previous results. Then, reintroduce changes one at a time to validate them. This helps you determine which change caused your model’s validity to deteriorate.
- Break the model down into parts that are easier to validate – If you have not already done so, validate simpler versions of this model. Remove components that you consider non-essential to reach a point at which you feel the results make sense. With fewer objects in the system, you will also have fewer parameters to configure and less room for error. Then, build back up again to a larger model, validating each step along the way.
- Question all of your assumptions – It is at this time that you will need to remember all of the important assumptions you have made. Assumptions are either made about how the system really works, or about the minimal impact of a particular simplification. In a component-oriented modeling environment like Modeler, your assumptions might be reflected in which model library objects to use and how to set their parameters. You might also have made some assumptions about background traffic or imported packet traces. All of these inputs to the model should be re-examined, particularly those that you feel could impact the validation significantly.
- Validate the validation – It is possible that instead of your model being flawed in some way, it is the experiment itself that is flawed. A measurement might have been taken incorrectly, or misinterpreted. If you are comparing against another model or mathematical calculation, that model or calculation might be in error.
Exact Results and Useful Results
While highly accurate results are generally desirable, it is important to emphasize that validation does not always mean an exact matching of results between your simulation and your measurements, or whichever other form of comparison you choose to use. Discrepancies are to be expected and can be accepted, provided that you feel you understand them and are in control of them. In other words, you should be able to account for differences and understand their importance relative to the questions you are trying to answer.
One common approach where exact results are not emphasized is sensitivity analysis. Here the objective is to vary one of the inputs of the model (the average size of a response returned by an application server, for example) and determine its effect on a model output (for example, the application response time). In such a case, you might be more concerned with the relationship between these two quantities, rather than whether they are exactly correct. In other words, is the response size a dominant factor in determining response time, or are other factors more important? If, for example, you determine that the application’s response time varies very little over the possible range of response sizes, you can focus your efforts on investigating other parts of the system.