Discrete Event Simulation Issues
Running simulations is typically thought of as the next-to-last step in the simulation and modeling process, the last step being results analysis. However, simulation is typically done many times in the modeling process. Individual simulations are used to validate and experiment. Finally, you will typically run a series of simulations to explore different issues and alternatives with your model. As you begin to run simulations you will be confronted with a few important issues concerning the configuration of your simulation runs.
This section describes the following issues:
- Simulation Duration
- Randomness
- Number of Simulations To Run
- Collecting Statistics
- Platform Variations
Simulation Duration
One of the most common questions concerns selecting a duration for the simulation. Duration refers to the amount of time simulated, not to the amount of wall-clock time that it takes to complete the simulation. We generally refer to wall-clock time as real-time, and time modeled in the simulation, as simulation time. Determining the duration of a simulation is typically driven by one of two factors: how much activity to simulate to obtain valid results; or simply, the time-span of activity that is required. The former requires some explanation, while the latter is obvious. Note, however, that you should always ensure that sufficient time has elapsed to allow representative activity to have occurred in your simulated model, even if this means simulating a longer period than you are required to by your project’s specification.
Randomness
Activities that have to be simulated are generally known to occur at a certain frequency or average rate. For example, if you are simulating users submitting transactions at a rate of 1 per minute and you want to obtain data on at least 30 transactions for each user, then it is a simple matter to choose a minimum simulation duration of 30 minutes.
It is important to realize that most simulations involve an element of randomness. It is possible to configure the traffic sources in the simulation to generate application requests or other forms of traffic in a perfectly regular pattern. However, this is typically not the case: instead, many parameters are configured with probability functions that characterize the likelihood that a particular outcome would occur. These probability functions are often referred to as PDFs, which stands for probability density functions.
For example, the time required for a server to generate a response to a client, given a request varies each time that it is measured. Given enough measurements at the same time of day, on a typical day, we can construct a PDF for Server Response Generation Time. Modeler in fact provides you a tool to do this using the Import Packet Trace feature, which will be described later. This PDF can then be used to tell server objects how to model response generation timing. Because response generation time is slightly different and that other parameters, such as response size and request size might also be variable, overall transaction response time, which is a function of all these, can also fluctuate. This is identical to the variations in real systems: each measurement is different.
Because the statistic we are trying to analyze may be subject to fluctuation, we cannot completely trust the results of one simulated activity, such as a transaction in the example above. Certainly, one measurement tells us something, namely that such a value is possible. This can be useful information in the sense that one particular measurement might be higher than what we consider acceptable and that therefore the proposed design must be revisited. However, in general, we want to obtain a more complete picture by looking at a collection of samples for the quantity of interest. So, we would collect many transaction response times, either by running a series of simulations or simply by letting the simulation run longer so that an application could regenerate another transaction at a later time.
Number of Simulations To Run
How many total samples are needed is a difficult question to answer. One of the reasons it is difficult is that this depends on the variability of the quantity of interest. Intuitively, this makes sense—if you obtain ten samples between 26 and 28 with an average of 26.5, you have a strong sense that this is the range you can expect for additional simulations, if you were to run them. Therefore, you can be confident in predicting the range and average with just ten samples. However, if the same ten samples varied between 20 and 40 with values widely scattered between these two bounds, you would feel uneasy about not obtaining further information.
Confidence can be established for a set of gathered samples by extracting a range from the samples and giving a likelihood that the real average lies somewhere in that range. By real average we mean the value we would obtain if we measured vast numbers of samples in the real system and computed their average. So, for our example, we might make a statement such as: From simulations, we have 95% confidence that the actual average response time is between 26 and 28 seconds. Again, this way of characterizing confidence makes intuitive sense because we cannot ever be certain that the samples we obtained from simulation were actually representative: they could all be exceptional values (so-called fluke values); however, the chance that this would be the case diminishes quickly as we gather more samples. Therefore, gathering more samples enhances our confidence, but never provides a 100% guarantee. To increase our certainty, we can increase the range we are seeking for the average value or we can obtain more samples.
While you can rely on your intuition to tell you if you have obtained enough samples, or some rules of thumb (many practitioners recommend the number 30 as a good number of samples), there are mathematical methods for determining confidence. Modeler helps you apply these methods by supporting the calculation of confidence intervals. These can be obtained in one of two ways:
- The Statistic Info operation provided for each graph allows you to see confidence intervals for 80%, 90%, 95%, 98%, and 99% confidence levels for the collection of points presented in the graph. See the important caveat below, however.
- The Show Confidence Intervals operation graphically displays confidence intervals, but assumes your data was collected using the scalar statistic collection approach, based on running many simulations. It generally does not apply to graphs that have time as their horizontal axis label.
When using confidence intervals, there is an important fact to be aware of concerning the relationship between the samples. These samples must be independent, which informally-speaking means no pair of samples must influence one another’s values or be linked indirectly to another condition that influences both of their values. In other words, the fact that the first sample was obtained should not influence the likelihood that the second sample would have been obtained. Such dependencies can occur in systems where state plays an important role. The following examples, still based on application response time, will help you understand when you can consider your samples to be independent:
- One client is connected through a network to one server and requests are submitted one at a time via the network to the server. The application response time is measured each time a response is returned to the client. Successive response times are measured for 25 transactions. These samples are independent because they have no influence over each other. As each transaction completes, the system returns essentially to the same state it was in prior to the transaction being initiated. You should get a similar collection of results running individual simulations with different random seed values and obtaining one sample per simulation.
- Many clients are connected via a network to one or more servers and submit requests to those servers. As in the previous example, a client’s current transaction must complete before it can submit another one. Response times for successive transactions are measured at a particular client. These samples are independent even though there is concurrent activity in the network, because the successive experiments still do not influence each other (or at least probably not significantly if they are sufficiently spaced out in time). The activities of other clients can be aggregated together and viewed as forming the statistical nature of the delay on the server. The fact that a first transaction was submitted by the client of interest does not significantly affect the response time for the second transaction it submits.
- The configuration is the same as in the prior example; however, we are now obtaining our samples from all clients. We may obtain hundreds or thousands of samples. So, the only difference is the fact that we are including more measurements from many locations. In this case, the samples are not independent because consecutive samples are reported by different clients that have influenced each others’ behavior. Specifically, if client A and B submit transactions at approximately the same time and finish at approximately the same time, both will record a result that is different than if they had submitted transactions separately. This is true because of contention for processor resources, which is modeled on the server. Therefore, you cannot believe the confidence intervals provided by the Statistic Info. operation.
Finally, if you want to use confidence interval measurements with the statistics provided by Modeler, read the following section on statistic collection for important information on how to configure statistics of interest prior to simulation.
For more information about confidence levels in simulation results and the value of running multiple simulations, see “Stochastic Modeling” on page 393.
Collecting Statistics
Before running a simulation, you will generally specify which statistics you want to collect. Modeler does not automatically collect all statistics in the system because there are so many available that you may not have enough disk space to store them. Specifying statistics is a straightforward task which is performed through the GUI, and this is not a time-consuming task in general.
There is one important subtlety associated with statistic collection, however. This has to do with how the reported statistics are processed by the simulator prior to being stored in the simulation results file(s). By default, nearly all statistics are gathered in what is called a bucket mode. Bucket refers to the fact that the statistic’s values are grouped and processed to reduce the number of samples reported in the statistic over the course of the simulation. This greatly reduces the amount of disk space required per statistic, which typically allows you to collect as many different statistics as you desire.
You can think of the statistic as having an original set of values that are driven by appropriate events in the simulation. The simulator then groups consecutive samples that occur within a time interval, processes them together, and presents them as one value. The time interval is the bucket, and the width of that bucket (measured in seconds) can be controlled by the user. The simplest way to control this parameter is to change it for all statistics by setting the Values per Statistic parameter of the Configure Simulation dialog box before running a simulation. This implicitly determines the amount of time spanned by each bucket because it divides the simulation duration (also specified in the same dialog box) into a fixed number of intervals.
Continuing with the example of application response time measurement, groups of values would correspond to not one, but many individual transactions completing (that is, returning completely to the client). Specifically, the response times for all the transactions that occurred within buckets of, say 10 seconds, would be averaged together to plot just one value in the statistic’s graph. Similarly, utilizations for links or PVCs are reported as the average utilization within a fixed-size bucket, not as instantaneous values. In general, bucket-oriented presentation of statistics is familiar and meaningful to most users, as this is the way information is often presented by network monitoring or reporting tools. However, there are cases in which you will want alternative ways of collecting and presenting data, as described in the remainder of this section.
In addition to controlling bucket width for all statistics together, you can also control the bucket width and other aspects of statistic collection for each statistic individually. You can do this by right-clicking on the statistic of interest in the Choose Results dialog box and selecting the Change Collection Mode operation. The default configuration of the statistic appears as Total of <default> Values. This means that the setting provided in the Values per Statistic parameter is to be used. You can choose a different number of samples, or you can use the advanced mode of this dialog box to choose other collection modes (also called capture modes) if you prefer not to use bucket-oriented collection.
Of particular interest is the All Values mode. This mode is important because it allows you to remove the effect of averaging or other processing performed in the bucket-oriented collection modes. Therefore, you can observe measurements for individual transactions, for example, instead of seeing the averaged results of many transactions that occurred within a bucket. This helps you gain a better understanding of what is actually happening within your simulation when you need to. It is particularly recommended to use this mode for some statistics during validation because you will gain more insight into the behavior of the model. Finally, using the All Values mode is essential if you want to apply the approach described above for measuring confidence and determining if you have simulated a sufficient amount of the system’s activity.
Sometimes, what you will see using the All Values mode will surprise you. Consider a utilization statistic, for example. Utilization consists fundamentally of an averaging of an on-off type of process. In other words, a link is either not in use or in use to transfer data at a given moment. To provide more insight into how the link is being used over periods of time, Modeler performs averaging over buckets. But when you look at the low-level information using All Values, you will see a stream of ones and zeroes. Furthermore, you will probably record many more points, so you should use the All Values collection mode cautiously, applying it only to a small number of statistics at a time. As a second example, consider a throughput statistic, measured in bytes per second. The normal processing mode for a throughput statistics is a bucket mode using a Sum / Time calculation, meaning that all of the values are added up and divided by the width of the bucket. Using the All Values mode in this case will display a series of packet sizes measured in bytes. Therefore, you will observe the sizes of the individual packets as they are processed.
For more information about other modes of statistic collection, see “Statistic Collection Mechanisms: Probes” on page 378.
Platform Variations
Operating system implementations can produce slightly different, but equally valid, results. For example, you might see a difference in the values of collected statistics or the number of events in a discrete event simulation when run on a Linux 32-bit architecture versus 32-bit Windows or a Linux or Windows 64-bit architecture. This is caused by different floating-point precision used for intermediate results. 32-bit Linux uses a 64-bit mantissa, whereas the other platforms use a 53-bit mantissa. This difference can affect results in DES, Flow Analysis, Transaction Analyzer, and any other operations that use floating-point math.