Phases of Data Collection

Models of complex systems typically evolve through stages of increasing complexity. As the model developer progresses, new features are added to the model until it converges on the desired level of realism. In the earlier stages, model developers typically run simulations for the purpose of verifying the specifications they have completed so far. In general, verification involves checking that some or all of the following criteria are met, as listed in the following table.

Table 113: Criteria Considered for Verification during Early Modeling Phases

Observable Criteria
Progress Simulation progress is the most basic criteria and is always examined first, often in an implicit manner by analyzing the remaining criteria. The model developer must of course verify that the simulation can be assembled and executed, and that it can process events, and finally reach completion without encountering serious errors. The number of events executed must appear reasonable; that is it must appear that there is meaningful generation and processing of events, and also that the density of events over simulated time is not excessive (simulations that have too great a density of events appear to advance very slowly).
Flow of Data Basic flow of packets in the system model is analyzed to ensure that packet generation, connections, and links are properly configured and that there are no unexpected bottlenecks or packet “sinks” in the system.
Basic Statistics Certain basic metrics in the system are understood in advance, or the model developer may have an intuitive notion of the range that the statistic should fall in. A subset of these statistics may be observed in the simulation to determine if the model has generally valid behavior.
Key Events If particular events are expected to occur at known times, or if particular processing is expected when certain events occur, the model developer may want to specifically analyze activity of the model surrounding those events to ensure correctness.

Because the model developer typically needs to do this type of work on a fairly frequent and repetitive basis, it is important that the observables of interest be quickly and conveniently accessible. Modeler provides features that directly support these types of verifications, as described in subsequent sections.

As a modeling project reaches later stages, model developers begin to focus on extracting information from the simulation that is specifically related to the goals of their project. This may of course include the items appearing in the table above. In addition, there are usually very specific questions to answer, as described in the following table.

Table 114: Information Collected in Later Modeling Phases

Observable Details
Application-Specific Statistics Each modeling project typically seeks to determine specialized metrics to answer the questions of system designers. These metrics may have to be obtained from customized computations provided by the modeler. Numerical post-processing of statistics may also be required after simulations complete.
Behavioral Characterizations A modeling project may also seek to characterize system behavior in a non-numerical manner. For example, particular sequences of events or correlation of events may be of interest to report.
Application-Specific Visualization Dynamic visualization can often provide a better understanding of the system’s activities. In some cases, this may require customized animation during simulation.