Generating Failures and Recoveries

Because a node or link cannot autonomously fail or recover, an external action must take place to change the state of the node or link during simulation. Typically, one of four such methods for generating failures and recoveries is utilized: deterministic, scripted, stochastic or interactive. The deterministic method causes one node or link to fail and recover at specific times during simulation. The script method is more general than the deterministic method and is based on a file that contains explicit times and locations of failures and recoveries. The stochastic method generates failure and recoveries based on a specified distribution of times and locations. With interactive manipulation, a failure or recovery is directly set through the debugger. Each of these methods is presented later as an example and may be modified to fit particular modeling requirements.

A failure or recovery occurs for an object when the condition attribute of the object changes value. Therefore, each of the four methods provides a mechanism to change the attribute’s value. However, they are different in the manner in which the object and the time for the failure or recovery is chosen.

Each of the methods, except interactive manipulation, requires that a process change the condition attribute of the node or link. Generally, one centralized process or many decentralized processes are responsible for generating the failures and recoveries. A centralized process typically resides in a node set aside only for this task and which would itself never fail. On the other hand, each decentralized process usually resides within a different node and is responsible for its parent node’s failure and recovery.

If a node is disabled, no interrupts will be delivered to the modules within the node. Due to this fact, a process within the node cannot directly cause the node to recover. To be invoked, the process itself requires an interrupt before it can change the node’s state, but it cannot receive any interrupts because the node is disabled. Therefore, if a process is responsible for causing its parent node to recover, the process may schedule a procedure call interrupt that will invoke a procedure that will cause the node to recover. Although a procedure call interrupt invokes the procedure within the context of the process that scheduled it, the interrupt is not delivered to an object within the node and, therefore, the Simulation Kernel does not prevent the invocation of the procedure. The diagram in the next subsection contains example code fragments from a process that causes its parent node to fail and then recover with this technique.

Deterministic Process Example

The deterministic process example presented in this section is based on the concept that a particular node or link fails and recovers at a known simulation time. The process has two attributes to specify the failure and recovery times: “fail time” and “recover time”. Optionally, the process could have another attribute that would specify the object to be modified, using, for example, a node’s or a link’s name. However, this example process simply causes its parent node to fail and then to recover, by setting the parent node’s condition attribute to “disabled” and then to “enabled”. Because this process is responsible only for its parent node, it can be considered an example of a decentralized failure/recovery process. The deterministic process method is recommended when simple failure/recovery modeling is required for the system. Typically just one failure/recovery cycle is scheduled, rather than a sequence of them. The following figures contain example code fragments of a deterministic failure/recovery process.

Figure 193: Example of Scheduling Failure and Recovery of Parent Node

/* Obtain the module’s and parent node’s object IDs. */
module_id = op_id_self ();
node_id = op_topo_parent (module_id);
/* Obtain the failure and recovery simulation times. */
op_ima_obj_attr_get (module_id, "fail time", &fail_time);
op_ima_obj_attr_get (module_id, "recover time", &recover_time);
/* Schedule the node’s failure and recovery via "call" interrupts. */
op_intrpt_schedule_call (fail_time, FAIL_CODE, fail_recov_proc, OPC_NIL);
op_intrpt_schedule_call (recover_time, RECOV_CODE, fail_recov_proc, OPC_NIL);

Figure 194: Example Function Block Procedure that Enables and Disables Parent Node

fail_recov_proc (state_ptr, code)
	char*  state_ptr;
	int    code;
	{
	Objid nd_objid;

	/** This procedure causes the parent node of the module that this **/
	/** interrupt was scheduled in to either fail or recover,         **/
	/** depending on the code.                                        **/
	FIN (fail_recov_proc (state_ptr, code))

	/* Obtain the parent node’s object ID. */
	nd_objid = op_topo_parent (op_intrpt_source ());

	/* Depending on the code, either enable or disable the node. */
	if (code == RECOV_CODE)
		op_ima_obj_attr_set (nd_objid, "condition", OPC_BOOLINT_ENABLED);
	else
		op_ima_obj_attr_set (nd_objid, "condition", OPC_BOOLINT_DISABLED);

	FOUT

Script File Process Example

A script file is a text file that specifies the failure and recovery times as well as the objects that are affected. Based on the script file, an associated process generates node and link failures and recoveries system-wide. Because this is the only process that causes failures and recoveries for the entire system, it can be considered an example of a centralized failure/recovery process. The script file process method is useful when a large or complex series of failures and recoveries occur during a simulation at specific times.

In this example, a script file contains a line for each failure or recovery event. Each line has three values: the time at which the event is to occur, the user ID of the object, and the type of event (failure or recovery). The user ID of the object is used rather than the object ID, because the “user id” attribute values of nodes and links are part of the network model and, therefore, are known in advance of the simulation. Object ID’s are assigned by the Simulation Kernel at run-time and can only be discovered during simulation. Figure 195 shows an example script file.

A script file may be generated from one of several sources. If the network model corresponds to an existing network, the script file may be based on an actual listing of node and link failures and recoveries. Another option is to create a script file that represents some specific system behavior, such as a series of nodes failing. A script file may also be used to test “worst case” scenarios, for example, where a set of interdependent nodes and links fail simultaneously.

The associated process reads the script file and builds a list of node and link failure and recovery times. After the list is built, the process schedules a self interrupt for the time of the first failure or recovery event. When the self interrupt occurs, the process sets the specified object’s condition attribute to the correct value by invoking the Kernel Procedure op_ima_obj_attr_set() and schedules another self interrupt for the next failure or recovery event. The following figure contains example code fragments of a script-based failure/recovery process.

Figure 195: Example Script File

#	Example script file for node/link failures/recoveries
#	Times are expected in order
#
#	time   user_id   event
	10.00    12      failure
	12.30    12      recovery
	14.40    3       failure
	14.40    4       failure
	14.45    5       failure
	14.55    4       recovery
	100.00   3       recovery

Figure 196: Example Proto-C Causing Object to Fail or Recover

/* This process just received a self intrpt that signals a failure */
/* or recovery action. Obtain the top instruction from the list.    */
inst_ptr = (Script_Inst*) op_prg_list_remove (event_list_ptr, OPC_LISTPOS_HEAD);

/* Convert user ID to object ID. */
object_id = user_id_to_object_id_convert (inst_ptr->user_id);

/* Set the object’s “condition” attribute to the specified value. */

if (inst_ptr->type == FAIL_CODE)
	op_ima_obj_attr_set (object_id, “condition”, OPC_BOOLINT_DISABLED);
else op_ima_obj_attr_set (object_id, “condition”, OPC_BOOLINT_ENABLED);

/* Deallocate instruction. */
op_prg_mem_free (inst_ptr);

/* Transition to next state, where next interrupt is scheduled. */

Figure 197: Example Proto-C Scheduling Failure/Recovery Self Interrupt

/* Obtain the next instruction from the list. */
inst_ptr = (Script_Inst*) op_prg_list_access (event_list_ptr, OPC_LISTPOS_HEAD);

/* Schedule a self interrupt for the time. */
op_intrpt_schedule_self (inst_ptr->time, 0);

Figure 198: Example Proto-C Reading Script File

/* Obtain name of script file from the process’ attribute. */
op_ima_obj_attr_get (op_id_self (), "script name", script_name);

/* Create and init list to hold failure/recovery instructions. */
event_list_ptr = op_prg_list_create ();

/* Load the file contents into a list. */
line_list_ptr = op_prg_gdf_read (script_name);

/* Loop through the lines of the list and parse each line. */
list_size = op_prg_list_size (line_list_ptr);
for (i = 0; i < list_size; i++)
	{
	/* Obtain the i_th line in the file. */
	string = op_prg_list_access (line_list_ptr, i);

	/* Decompose the line in to fields. */
	field_list_ptr = op_prg_str_decompose (string, " ,/\t");

	/* If the line does not have three fields or starts with '#' */
	/* then skip it and try the next.                            */
	if (op_prg_list_size (field_list_ptr) != 3)
		goto dealloc;
	field0 = op_prg_list_access (field_list_ptr, 0);
	if (field0 [0] == '#')
		goto dealloc;

/* Allocate a structure to hold the event instruction. */
	inst_ptr = (Script_Inst*) op_prg_mem_alloc (sizeof (Script_Inst));

	/* Fill in the instruction. */
	inst_ptr->time = atof (field0);
	inst_ptr->user_id = atoi (op_prg_list_access (field_list_ptr, 1);
	field2 = op_prg_list_access (field_list_ptr, 2);
	if (strcmp ("failure", field2) == 0)
		inst_ptr->type = FAIL_CODE;
	else inst_ptr->type = RECOVER_CODE;

	/* Place the instruction on the end of the list. */
	op_prg_list_insert (event_list_ptr, inst_ptr, OPC_LISTPOS_TAIL);

	/* Deallocate the elements in the list and then the list itself. */
	dealloc:
	op_prg_list_free (field_list_ptr);
	op_prg_mem_free (field_list_ptr);
	}

/* Deallocate the elements in the list and then the list itself. */
op_prg_list_free (line_list_ptr);
op_prg_mem_free (line_list_ptr);

Stochastic Process Example

A stochastic failure/recovery process generates node and link failures and recoveries based on distributions, so that the states of the nodes and links change randomly. The example process has four attributes that determine the frequency of the failure and recovery events. The “node fail mean” and “link fail mean” attributes specify the mean failure times for a node and a link, respectively. Similarly, the “node recover mean” and “link recover mean” attributes specify the mean recovery times. The process chooses a particular node or link randomly for a failure and then schedules a recovery for that node or link. Often, the exponential distribution is used to model the failure and recovery times. However, other distributions may be more accurate for a particular system.

A stochastic process is recommended when randomly generated failures and recoveries are an appropriate model for the system. A stochastic process is typically more practical than a script file process when a large number of failures and recoveries occur. A stochastic process has the additional benefit that it dynamically generates failures and recoveries, adjusting for different size networks and simulation durations. Also, it can automatically subject a network to typical scenarios without requiring new user input.

The example process generates failures for nodes and links in parallel. Initially, the process obtains the mean failure and recovery attribute values. Two failure times are computed based on the mean times and one self interrupt is scheduled by the process for a node failure, and another for a link failure. When a self interrupt occurs for a failure, the process chooses a node or link at random and causes it to fail by setting its condition attribute to “disabled” via the KP op_ima_obj_attr_set(). The process then computes a recovery time for the node or link based on the specified mean recovery time and schedules a self interrupt for the recovery and includes the node’s or link’s object ID as the code of the interrupt. The object ID is included so that the process can identify which object to enable. When this interrupt occurs, the condition attribute of the object is set back to “enabled”. Along with scheduling an interrupt for the recovery, the process also schedules the next failure self interrupt. The following figure contains code fragments of an example stochastic failure/recovery process.

Figure 199: Example Proto-C Causing Object to Fail or Recover

/* This process just received an interrupt.  Determine what is to  */
/* happen.  Node and link failures use the codes; recovery intrpts */
/* contain the object ID of the object to recover.                 */
code = op_intrpt_code ();
if (code == NODE_FAIL_CODE))
	{
	/* Obtain random node object ID and determine if already disabled. */
	object_index = (int) op_dist_uniform (num_nodes);
	object_id = op_topo_object (OPC_OBJMTYPE_NODE, object_index);
	op_ima_obj_attr_get (object_id, “condition”, &current_condition)
	if (current_condition != disabled)
		{
		/* Cause the node to fail. */
		op_ima_obj_attr_set (object_id, “condition”, OPC_BOOLINT_DISABLED);

		/* Schedule recovery interrupt; first generate recovery time. */
		intrpt_time = op_sim_time () + op_dist_outcome (node_recov_dist_ptr);
		op_intrpt_schedule_self (intrpt_time, object_id);
		}

	/* Schedule the next fail interrupt; first generate failure time. */
	intrpt_time = op_sim_time () + op_dist_outcome (node_fail_dist_ptr);
	op_intrpt_schedule_self (intrpt_time, NODE_FAIL_CODE);
}
else if (code == LINK_FAIL_CODE)
	{
	/* Obtain random link object ID and determine if already disabled. */
	object_index = (int) op_dist_uniform (num_links);
	object_id = op_topo_object (OPC_OBJMTYPE_LINK, object_index);
	op_ima_obj_attr_get (object_id, “condition”, &current_condition)
	if (current_condition != disabled)
		{
		/* Cause the link to fail. */
		op_ima_obj_attr_set (object_id, “condition”, OPC_BOOLINT_DISABLED);

/* Schedule recovery interrupt; first generate recovery time. */
		intrpt = op_sim_time () + op_dist_outcome (link_recov_dist_ptr);
		op_intrpt_schedule_self (intrpt_time, object_id);
		}

	/* Schedule the next fail interrupt; first generate failure time. */
	intrpt_time = op_sim_time () + op_dist_outcome (link_fail_dist_ptr);
	op_intrpt_schedule_self (intrpt_time, LINK_FAIL_CODE);
	}
else
	{
	/* This is a recovery; the code is the object ID of the object */
	/* that is recovering.  Set the “condition” attribute.         */
	op_ima_obj_attr_set (code, “condition”, OPC_BOOLINT_ENABLED);
	}

Interactive Manipulation

When a simulation is run under the OPNET simulation debugger, attributes of objects can be set directly with the debugger command attrset. Through this mechanism, a node or link condition attribute can be set to “enabled” or “disabled”. If the value changes, recovery or failure interrupts will be generated exactly as if some process had changed the attribute’s value via a call to the Kernel Procedure op_ima_obj_attr_set(). Interactive manipulation is useful for testing the system’s reaction to failures and recoveries “on-the-fly”.

本站文档由 北京智网畅联科技有限公司 整理、翻译并提供 · Documentation compiled and provided by Beijing ZNetLink Technology Co., Ltd.

交流群 · QQ Group:OPNET/智网妙算网络仿真交流群(群号 486383494)

原文档 © 2017 Riverbed Technology · 本站为公益翻译,由 AI 辅助完成,请仔细甄别

Original documentation © 2017 Riverbed Technology. Community translation, AI-assisted. Contact: jiazequn@znetlink.com