Working with the Parallel Simulation Kernels
To safely run model code in parallel, model developers must follow some modeling techniques and conventions that differ from those for sequential simulations.
Because parallel modeling techniques and conventions apply to the technical implementation and not the modeling approach, they have little impact on overall model design. The parallel kernels can run parallel-safe and legacy models in the same network. This allows model developers to declare their model code parallel-safe in phases.
Note: In this context, “parallel safe” refers only to the ability of code to operate safely in an Riverbed parallel-simulation environment. This is not equivalent to operating in user-created threads with an OS thread manager. You cannot call Riverbed functions (KPs or otherwise) from user-created threads.
A phased approach enables developers to achieve higher performance selectively, without having to modify the entire model suite at once. Model code that has not been specifically reviewed and modified to take advantage of the parallel simulation kernels can still be executed safely. However, only models that have been declared parallel-safe will benefit from the parallelism achieved in the kernels.
In this section, “model code” refers to the process model, pipeline stage, packet format converter, and external C-code modules that have been created by the model developer or are part of the Standard Model Library. Unless otherwise noted, the models in the Standard Model Library have not been made parallel-safe.
Model code is parallel-safe if it can run concurrently with other parallel-safe model code in a multi-threaded environment and produce correct results. For example, code that reads and increments a global variable is not parallel-safe—if several instances of the model code are running simultaneously, the unexpected ordering of execution can cause incorrect values to be stored in the global variable.
Updating existing model code for parallel execution (making code parallel-safe) is a three-phase process:
- Identify sections of model code that are not parallel-safe, modify them using the techniques described in this section, then validate the results by running parallel and sequential simulations.
- Profile parallel-simulation runs to expose inefficiencies specific to the parallel environment, such as serial execution of model code that exhibits high contention. This is not strictly necessary for correctness, but is an important aspect of parallelization to improve system performance.
- Redesign models (to avoid bottlenecks identified during the profiling stage) or change the flow of execution to achieve more calculations in parallel. Again, this improvement is related to performance and not model correctness and can be done at a later stage of development.
Declaring Pipeline Stages Parallel-Safe
The convention for functions declared in pipeline stages is that the “plain” function name (for example, dra_chanmatch) represents a non-parallel-safe pipeline stage; a parallel-safe pipeline stage has an _mt suffix in the function name (such as dra_chanmatch_mt). In both cases, the file name is the “base” name (thus, dra_chanmatch_mt is still found in dra_chanmatch.ps.c).
The _mt versions of pipeline stage functions must include the OP_SIM_CONTEXT optional context argument macro in their argument list.
Here is the code for a simple parallel-safe closure pipeline model:
#if defined (__cplusplus)
extern "C"
#endif
void
dra_closure_all_mt (OP_SIM_CONTEXT_ARG_OPT_COMMA Packet* pkptr)
{
/** Guarantee closure between transmitter and receiver. **/
FIN_MT (dra_closure_all_mt (pkptr));
/* Place closure status in packet transmission data block. */
op_td_set_int (pkptr, OPC_TDA_RA_CLOSURE, OPC_TRUE);
FOUT
}
The OP_SIM_CONTEXT_ARG_OPT_COMMA macro is used to denote the simulation context argument required for multithreaded variants of pipeline stages.
At simulation time, the parallel simulation kernel looks for both versions of the pipeline stage. The pipeline stage is considered parallel-safe only if the kernel finds the _mt version. If you declare both the “plain” and “mt” functions in a pipeline stage, you can use the same model file (.ps.c) to run parallel and sequential simulations.
Initializing and Terminating Simulations
To simplify initializing and terminating model simulations, all BEGINSIM and ENDSIM interrupts are executed sequentially by the parallel kernel regardless of whether the modules processing the interrupts have been declared parallel-safe. There is no need to worry about thread safety of code handling those two interrupts.
Similarly, all events at simulation time 0 are executed sequentially. Consequently, you can ignore race condition and non-determinism issues in model code when dealing with “init” states. However, one note of caution: child process models can be created dynamically and initialized after time 0, so do not assume that a process model’s init state is necessarily going to be executed at time 0.
Generating Random Numbers
The parallel simulation kernel does not guarantee that simulation events at the same simulation time are executed in a specific order. This means that the global random number generator invoked from parallel code is likely to generate a sequence of random numbers that differs from one model execution to the next. This is not an issue in the sequential simulation kernel because the order of event execution is deterministic.
To work around this issue, both the parallel and sequential kernels use the “per_module_random_stream” preference. If this preference is set to TRUE, each module produces an independent stream of random numbers unaffected by the numbers generated in other modules. This guarantees that, as long as events for the same module are scheduled in the same order, any op_dist KP calls will generate a reproducible sequence of random numbers. Per-module random number streams are helpful for debugging and validation because they allow simulations to produce the same results across different parallel and sequential runs of the same model.
When per-module streams are used, the value of the “seed” preference is used to create an initial random number sequence. The output of this initial sequence is used to generate random seeds for the independent random number sequences of various modules.
By default, per_module_random_stream is set to FALSE in the sequential kernel and TRUE in the parallel kernel. Support for individual random number streams on each module increases the amount of memory required for each module by about 200 bytes.
Serializing IDs
In the parallel kernels, event, packet, and ICI IDs are serialized by module rather than being based on a global counter. As a result, IDs generated by different modules are not comparable; only IDs generated by the same module are ordered. Events and packets created by the Simulation Kernel use a special module ID.
In ODB, module-serialized IDs are displayed as “<module_id>:<module_counter>” or as “kernel:<module_counter>”.
Preferences for op_runsim allow you to specify the serialization scheme desired. These preferences default to TRUE (global serialization) for sequential simulations and to FALSE (module serialization) for parallel ones.
- “fully_serialized_event_ids”
- “fully_serialized_event_execution_ids”
- “fully_serialized_ici_ids”
- “fully_serialized_packet_ids”
Compiling Code
Object files and libraries are not compatible between the parallel and sequential simulation kernels. Therefore, Modeler produces and uses distinct object files for each simulation kernel.
By default, command line utilities such as op_mko, op_mkso, op_mksim, and op_runsim produce object code suitable for the 32-bit sequential development kernel. The types of object files, libraries (repositories), or executables generated are based on the common “kernel_type” preference. The following table lists the values available for parallel simulations.
Table 129: kernel_type Preference Values for Parallel Simulation
| Preference Value | Description |
|---|---|
| development_parallel | 32-bit parallel development kernel |
| optimized_parallel | 32-bit parallel optimized kernel |
| development_parallel_64bit | 64-bit parallel development kernel |
| optimized_parallel_64bit | 64-bit parallel optimized kernel |
In the GUI, a set of preferences can be used to define for which kernels to compile a model. The following table lists these preferences for parallel simulations.
Table 130: comp_target_type Preferences
| Preference | Default Value |
|---|---|
| “comp_target_type.development_parallel_32” | FALSE |
| “comp_target_type.optimized_parallel_32” | FALSE |
| “comp_target_type.development_parallel_64” | FALSE |
| “comp_target_type.optimized_parallel_64” | FALSE |
The following preferences control compilation output:
Table 131: comp_flags Preferences
- “comp_flags_common” — Flags always passed through to the compiler. By default it is set to include the standard models headers directory in the compiler’s headers search path.
- “comp_flags_c_specific” — Used when compiling C code. Empty by default.
- “comp_flags_c++_specific” — Used when compiling C++ code. Empty by default.
- “comp_flags_devel” — Used when producing output for the development kernel. By default is set to produce output that includes native debugger code instrumentations.
- “comp_flags_optim” — Used when producing output for the optimized kernel. By default, this is set to use platform-specific compiler optimizations.
- “comp_flags_mt” — Used when producing output for the parallel kernel. Empty by default.
Before Modeler 10.0, each Kernel Procedure explicitly cast its arguments to the correct types expected for a KP. Although this was convenient for the developer, it was arguably a dangerous policy because it could obscure hard-to-detect programming mistakes. Beginning with Modeler 10.0, Kernel Procedures do not cast their arguments. Because some changes are likely to be made to models to make them parallel-safe, make sure you update model code to produce no compilation warnings.
Note: If you are using gcc under Linux, make sure that you compile model code with the “comp_prog” preference set to comp_gcc_paranoid. This generates compilation errors for suspicious pieces of code.