Making Models Parallel-Safe
The following sections describe the technical details of making models parallel-safe.
Mutexes
For code that should not be executed concurrently, you can use various serialization mechanisms to ensure that at most one thread accesses the code at a time. Mutual exclusion locks (mutexes) are a common way of forcing serialization by making sure that if multiple threads try to access a section of critical code, only one will be granted permission to proceed. The other threads must wait for the active thread to complete execution, at which point one of the waiting threads will begin processing the code.
Modeler provides a set of KPs for creating and using mutexes; these are described in “Mutex Sub-Package” on page 5. An additional package (prg_mt) is available for situations where finer control of serialization is needed; see “Mutex Package” on page 90 for details.
Mutex Serialization
The following diagram shows in detail how mutexes enforce code serialization. A thread requests access to a critical section of code by attempting to lock the mutex. When the lock completes, the thread can proceed. Other threads attempting to lock the mutex will be blocked. Once the thread that successfully locked the mutex has completed the critical section of code, that thread must unlock the mutex to allow other waiting threads to proceed.
Figure 258: Mutual Exclusion Locks (Mutexes) in Action

The same mutex can be used to guard multiple sections of code, such as the various functions of a list management package. This ensures that once a thread is performing one of the functions, no other threads can modify the internals of the list package.
The default mutex does not allow recursive locking by the same thread. If a thread that had locked a mutex were to attempt to lock it again, it would be blocked waiting for itself to complete. The locking KP can detect this situation and does not block; instead, it reports the potential problem in the code via an appropriate error message.
There are times when it is much simpler if a mutex allows any number of recursive locks. Such a mutex is appropriately named a recursive mutex. Each lock increments an internal counter. Each unlock decrements the counter. Only when the lock counter reaches 0 is the mutex available for another thread.
To use a mutex in your pipeline stages, you must ensure that the mutex has been created before any pipeline stages are invoked (for example, you must create the mutex before entry into the link closure pipeline stage). The mutex is then stored as a global variable accessible by the various pipeline stages.
Mutex Deadlocks
Using multiple mutexes can lead to deadlock situations in which a set of threads is blocked indefinitely. Each thread in the set has locked one or more mutexes and is waiting for the availability of one or more mutexes held by other threads in the set. None of the threads can complete the code to unlock any mutexes they have locked, and thus cannot allow other threads to proceed.
A simple case is shown below, with two threads trying to lock the same two mutexes, but using different locking orders:
Thread A – Lock (mutex1); Lock (mutex2); <Critical Section A>; Unlock (mutex1); Unlock (mutex2);Thread B – Lock (mutex2); Lock (mutex1); <Critical Section B>; Unlock (mutex1); Unlock (mutex2);
If both threads perform their first lock successfully, then thread A will hold mutex1 and wait for mutex2, while thread B will hold mutex2 and wait for mutex1, creating a deadlock situation.
Features in the design of the parallel simulation KPs help you detect and prevent possible deadlocks. See the description of op_prg_mt_mutex_create() on page 72 for more information.
Global Variables
Non-constant global variables are a major reason why model code is considered unsafe for parallel execution; race conditions caused by non-deterministic parallel execution during global variable updates can result in incorrect values.
For example, consider the following code:
int next_unique_id = 0;
int
my_unique_id_get (void)
{
int id = next_unique_id; /* A */
next_unique_id++; /* B */
return id;
}
If my_unique_id_get() were executed in parallel in two different threads and both threads executed statement A at the same time, the same number would be returned in both threads. This would break the uniqueness of the generated value.
Furthermore, the C-language post-increment and pre-increment operators (++) are not atomic; this means that several CPU instructions are used to implement them. As a result, the outcome of statement B is non-deterministic in the context of concurrent execution. To ensure deterministic behavior, access to change the global variable should be protected by a serializing construct such as a mutex, as in the following code:
int
my_unique_id_get (void)
{
int id;
prg_mt_global_mutex_lock ();
id = next_unique_id;
next_unique_id++;
prg_mt_global_mutex_unlock ();
return id;
}
Global variables that are modified only during simulation initialization (during a BEGINSIM interrupt, for example, or during time 0 of the simulation) do not pose a problem because these events are serialized by the simulation kernel and are guaranteed to be executed sequentially. This means the model code that reads global variables for the rest of the simulation does not need to be modified to serialize its access to them.
Global variables that are set only when they are needed for the simulation can also be efficiently protected with the following construction:
static void * set_once_variable;
static int set_once_variable_set = 0;
if (!set_once_variable_set)
{
/* Serialize execution */
op_prg_mt_global_mutex_lock ();
/* Check condition again: some other thread may have */
/* initialized the value */
/* while we were trying to lock the mutex */
if (!set_once_variable_set)
{
/* initialize */
set_once_variable = init_that_variable ();
/* Set global flag AFTER initialization is complete */
set_once_variable_set = 1;
}
/* Release the mutex */
op_prg_mt_global_mutex_unlock ();
}
Because there is some runtime overhead involved, it is best to set all such variables during the serialized initialization stage whenever possible.
Access to global variables that can be changed during simulation needs to be serialized by some mutex. Mutex serialization is significantly more expensive when there is a lot of contention (on the order of hundreds of times on some platforms). Contention occurs when two or more threads try to obtain the same mutex at the same time. For this reason, you should avoid using global mutexes. The fact that event execution is serialized on modules can help to avoid using mutexes in some cases. In other cases, partitioning objects into subsets of objects, with each sharing its own global mutex, can help. For example, it may be more efficient to have a set of objects that belong to the same subnet or same radio transmission group share a private mutex, instead of having groups contend for a global mutex.
Access to nonstatic variables declared in the temporary variables block of a process model do not need to be serialized; these are local to each instance of a process invoked to handle an event.
State Variables
For efficiency, access to state variables is not serialized. Therefore, KPs that access state variable values should not be used to access state variables from processes other than those associated with the module that owns the current process. In other words, it is safe to use those KPs to access state variables of the child, parent, or sibling processes associated with the same module as the current process. However, if model code accesses state variables of any other processes, these KP calls must be serialized through shared mutexes.
Prg Data Structures
The use of hash and list data structures require special attention in parallel simulations, as described in the following sections.
Prg_String_Hash
Additions, modifications, and iterations over string hashes are not parallel-safe and must be serialized. However, “read” access to static hash table data does not need to be serialized. In the very common case where data is calculated and inserted into a hash if it was not inserted before, the following construction is safe:
void *data_ptr = prg_string_hash_table_item_get (hash_ptr, key_string);
if (!data_ptr)
{
op_prg_mt_global_mutex_lock ();
/* check again (other thread may have inserted the data already */
data_ptr = prg_string_hash_table_item_get (hash_ptr, key_string);
if (!data_ptr)
{
data_ptr = calculate_data (key_string,…);
prg_string_hash_table_item_insert (hash_ptr,
key_string, data_ptr, NULL);
}
op_prg_mt_global_mutex_unlock ();
}
This construction is safe only if the modifications to the hash table and its contents are additions of new elements. That is, removing elements or modifying elements stored in the hash is not allowed.
Prg_List
List access is generally not parallel-safe. Access to all shared lists accessible from different processes must be serialized. Furthermore, iterations and element-access operations on shared lists are not parallel-safe. Therefore, dynamic list operations must be serialized through a mutex. You can consider replacing lists with arrays, because read access to arrays is parallel-safe; this is not the case for lists.
Simulation Context Pointer
To execute code efficiently, the parallel simulation Kernel Procedures require an additional opaque simulation context argument. Model code that invokes Kernel Procedures is required to pass this context pointer argument to those KPs. This means that user-defined code must be able to obtain this simulation context pointer so it can pass it to Kernel Procedures that it calls.
There are three ways to obtain the context pointer:
- Using a FIN macro (which is expected to be present in all functions before any code other than variable declarations appears). The macro implicitly obtains the correct simulation context and puts it into a local variable.
- A process model’s finite state machine gets the context pointer as one of its arguments. Therefore, any code in the enter or exit executives and transitions has access to this context pointer automatically.
- Parallel-safe pipeline stages have the simulation context passed to them explicitly as an additional argument.
FIN and FIN_MT Macros
The implicit lookup and assignment of the context pointer of the FIN macro is not very expensive (in terms of execution time). However, it may become a bottleneck for functions that are called frequently, such as comparators in search operations or frequently-called calculations over large data sets. In these situations, you can change the function declaration to include a simulation context pointer as one of its arguments and have callers of that function explicitly pass that context through. Because most calls to user-defined functions originate from the finite state machine or from pipeline stages, you can change all functions to take a simulation context pointer as an argument and propagate it to all model code.
There are two problems you might discover in existing model code:
- Your functions do not use the FIN/FOUT/FRET macros. The simulation context will be undefined in such functions unless it is explicitly passed to them.
- Because the simulation context pointer is required for access to process model state variables, state variables might not be accessible before the FIN statement.
For example, assuming SV is an integer state variable, the following code will not compile for parallel execution:
void my_proc(void)
{
/* Use FIN_MT since the context is explicit */
int j = SV;
FIN (my_proc);
…
You need to change the code as follows:
void my_proc(void)
{
int j;
FIN (my_proc);
j = SV;
…
To avoid declaring a duplicate simulation context and to eliminate the overhead of determining the current context, you can replace the FIN statement with FIN_MT in functions that were changed to take an explicit simulation context argument or functions that do not need the simulation context. This avoids the extra work that the standard FIN statement performs to obtain the implicit simulation context.
Context Macros
To help model developers maintain a code base of models that are suitable for both sequential and parallel execution, there are a few macros defined in the simulation kernel headers that can be used in all model code. These macros eliminate the overhead of passing around an unnecessary simulation context during sequential execution (when the context is not needed) by expanding to nothing. However, when the same model code is compiled for parallel execution, the macros provide the necessary functions for the required simulation context:
OP_SIM_CONTEXT_ARG_OPT
OP_SIM_CONTEXT_ARG_OPT_COMMA
These macros should be used in function declarations for functions that expect an explicit simulation context pointer argument passed by the caller during parallel execution. The variation with _COMMA has a comma included in the expanded parallel macro. Both macros expand to nothing for sequential simulation.
OP_SIM_CONTEXT_PTR_OPT
OP_SIM_CONTEXT_PTR_OPT_COMMA
These macros should be used to pass a simulation context pointer to a function that is expecting it as an argument during parallel execution. The variation with _COMMA has a comma included in the expanded parallel macro. Both macros expand to nothing in the sequential kernel.
The following example shows how to use these macros:
/* Function prototypes */
void
void_function (OP_SIM_CONTEXT_ARG_OPT);
void
function_that_takes_int_arg (OP_SIM_CONTEXT_ARG_OPT_COMMA int x);
/* Functions */
void
void_function (OP_SIM_CONTEXT_ARG_OPT)
{
FIN_MT (void_function);
/* call function with an argument */
/* The first argument is the simulation context itself */
/* and the second is the integer ‘23’. During sequential */
/* execution, there is only one argument passed in since the */
/* OP_SIM_CONTEXT_PTR_OPT_COMMA macro expands to nothing. */
function_that_takes_int_arg (OP_SIM_CONTEXT_PTR_OPT_COMMA 23);
/* here is how to call a void function that takes a sim */
/* sim context pointer as an argument. */
void_function (OP_SIM_CONTEXT_PTR_OPT);
FOUT;
}
Unsafe and Unsupported KPs
All Kernel Procedures supported by the parallel simulation kernel are assumed to be parallel-safe. However, the functions and data exposed by some KPs are inherently unsafe or dangerous in the parallel environment. Use of such KPs must be carefully examined. Modeler has not removed these KPs from the parallel kernel because they can still be used in models that are not marked as parallel-safe and are executed sequentially. Furthermore, if they are used carefully, these KPs provide convenience for model developers.
In addition, some KPs were seldom used in model code and were deemed too expensive and too dangerous to support in the parallel kernel. In brief, functions to support interrupt priorities and iterate over pending events have been removed from the parallel simulation kernel.
See “Multi-Threading Safety” on page 11 for a summary of unsafe and unsupported KPs.
Priority Attributes
In the parallel simulation kernel, the built-in “priority” attribute has been removed from nodes of all types and both the “priority” and “super priority” built-in attributes have been removed from all modules.