The RTR Manager provides access to commands and displays with which you manage RTR, its facilities, nodes, logs, and partitions. The RTR Manager also provides enhanced graphical monitoring capabilities superior to the RTR CLI. Rows are color-coded, hyperlinks are available, and popups describe the content of monitor screens. Most RTR CLI commands are also available through the RTR Manager screen when you click on the RTR Command link.

In addition, the browser interface provides context-sensitive help for commands where user input is required, informational popups on certain headings such as "Resource," and details of a facility, partition, link, or transaction with links through browser popups. For example, to see details of a defined partition, click on the "show partition x details" popup seen when placing your mouse on a blue-underlined partition name.

For information about server security and customizing the appearance of the RTR Administrator interface, see Appendix B, Customizing the RTR Web Browser Interface.

The RTR Explorer interface enables the system administrator to monitor all nodes in the facilities defined for RTR, to navigate to all nodes in the RTR network, drill down to specific nodes, and receive automatic notification of warning, error, or fatal alarms. Nodes are self-monitoring; they detect faults in RTR nodes, roles, links, or partitions.

When everything is normal, RTR Explorer shows all nodes in graphical displays, and presents an alert when the state of a node or facility changes to an abnormal state. Three alarm levels are shown in color and an increasing number of flags: Warning (yellow, one flag), Error (orange, two flags) and Fatal (red, three flags).

Figure 1.6, “RTR Explorer - Navigation Mode, All-Facility View” shows an all-facility view with several alarm states. Buttons on the right pane let you Go Back to Home (the RTR Administrator screen), Switch to Alarm Mode, Turn On (or Off) Refresh, and Display Help.

The mouse pointer is over the ‘‘Go Back to Home’’ button, showing instructions in the bottom left information pane.

To find which node is sending an alarm and information about the alarm, click on the facility you want to examine. For example, the result when you click on the ‘‘all_images_fac’’ facility at the top of Figure 1.6, “RTR Explorer - Navigation Mode, All-Facility View” is shown in Figure 1.7, “RTR Explorer - Alarm Screen”. This node-and-alarm view provides information about each node in the facility that has an alarm. For more information about RTR icons, see Section 1.3.2.1, “RTR Explorer Icons”. For information about RTR Explorer Help (viewed from the Display Help button), see Section 1.3.2.2, “RTR Explorer Help”.

1.3.2.1. RTR Explorer Icons

Explorer icons indicate states for nodes, facilities and partitions to assist you in finding and correcting the source of an alarm. Icons have a well-defined hierarchy from OK (green, no flags), through Warning (yellow, one flag) and Error (orange, two flags) to Fatal (red, three flags). The flags assist users who are color blind.

Several icon types assist your use of RTR:

• Alarm icons (flags only, on the alarm screen)

• Facility icons

• Node icons

• Node-group icons

• Group-by icons

• Partition-state icons (described in Chapter 3, Partition Management).

When you place your mouse pointer over an icon or button, an information pane appears (lower left) describing the object under the mouse pointer. More detail on icons is available in the RTR Explorer interface.

Facility Icons

The icon status of a facility corresponds to the most serious status in the facility. To view the nodes in a specific facility and their individual status, click on the facility icon. The facility icons are shown in Figure 1.8, “RTR Explorer Facility OK Icon” to Figure 1.11, “RTR Explorer Facility Fatal Icon”.

You obtain additional information to track down alarm origins using the node, node-group, and group-by-icons.

Node Icons

A node icon represents an individual node in an RTR facility; its status, shown by color and flags, represents the most serious alarm reported by that node. Node icons are shown in Figure 1.12, “RTR Node OK Icon” to Figure 1.15, “RTR Node Fatal Icon”.

Node-Group Icons

In a large configuration where node icons cannot fit on one screen, nodes are grouped alphabetically, and a group icon shows the status of each group. Nodename alphabetization is case-insensitive; for example, Y.Z.COM comes after a.b.com. Node-group icons are shown in Figure 1.16, “RTR Node-Group OK Icon” to Figure 1.19, “RTR Node-Group Fatal Icon”.

Group-By Icons

When examining a facility, you can group backends in several ways:

• With no group

• By cluster

• By partition

• By cluster, then partition

• By partition, then cluster

Use these groupings to help you isolate the source of an alarm, as shown in Figure 1–20.

Group-by icons are shown in Figure 1–21 to Figure 1–25. Icons help remind you which grouping is shown, as follows:

Partition State Icons

Partition states show the current state of a given partition; they appear on the facility screen. They are described in Chapter 3, Partition Management.

% rtr help

% rtr help show

% rtr help errors error-identification

where error-identification identifies the returned error.

% rtr
RTR> start rtr
%RTR-F-RTRALRSTA, rtr already started
RTR> help errors rtralrsta
  
Errors
  
  RTRALRSTA
   
RTR already started
   
    Explanation: RTR was already running when the "START RTR" command
    was executed. This error message is displayed by the RTR utility.
   
RTR>

% rtr 
 RTR>  start rtr 
 
 RTR>  create facility funds_transfer/frontend=(FE1,FE2,FE3) -
 _RTR>                            /router=(TR1, TR2) - 
 _RTR>        /backend=(BE1, BE2, BE3)

Frontends and backends need to know about the routers to which they are connected, but frontends don't need to know about backends or other frontends, nor backends about frontends or other backends. In creating facilities, RTR ignores superfluous node names so that you can use the same CREATE FACILITY command on every node in an RTR configuration. This helps to simplify maintenance of startup command procedures.

For example, the same RTR CREATE FACILITY command shown in Example 2.2, “Remote Setup from One Node” can be issued on all nodes in the configuration. When issued on FE1, RTR ignores the list of backends because nodes only need to know about the nodes in the neighboring layers of the configuration, thus FE1 does not need to know about BE1.

 % rtr
 RTR>  set environment/node=(FE1, FE2, FE3, TR1, TR2, BE1, BR2, BR3)
 RTR>  start rtr
 RTR>  create facility funds_transfer/frontend=(FE1, FE2, FE3) -
 _RTR>                           /router=(TR1, TR2) -
 _RTR>                           /backend=(BE1, BE2, BE3)

This enters the commands on every node in the configuration. You must have an account with the necessary privileges on any node where the command is to execute.

Use the SHOW RTR command to find out if RTR has been started on a particular node.

RTR> CREATE FACILITY OSIRIS/backend=NODEA/router=NODEA/frontend=1x.2x.xx.xxx
%RTR-S-FACCREATE, facility OSIRIS created
RTR> SHOW FACILITY...

Both the /LINKS and /PROCESSES qualifiers have high default values.

The default value for /LINKS is 512. This value is high but is chosen to protect RTR routers against a failover where the number of frontends is large and the number of surviving routers is small. The maximum value for /LINKS is 1200.

The default value for /PROCESSES is 64. This value is large for frontend and router nodes but is sized for backends hosting applications. Backends with complex applications may have to set this value higher.

The maximum value for /PROCESSES is the OpenVMS allowed maximum. Warning messages are generated if the requested (or default) memory quotas conflict with the system-wide WSMAX parameter, or if the calculated or specified page file quota is greater than the remaining free page file space.

The default values for /LINKS and /PROCESSES require a large page file. RTR issues a warning if insufficient free space remains in the page file to accommodate RTR, so choose values appropriate for your configuration.

The /LINKS and /PROCESSES qualifiers do not take into account memory requirements for transactions. If an application passes a large amount of data from client to server or vice-versa, this should be included in the sizing calculations. For further information on the START RTR qualifiers, see the START RTR command in the Command Reference section.

You may explicitly specify values for the VMS process quotas listed above, but it is an error to specify a value smaller than that calculated by RTR based on the values supplied with /LINKS and /PROCESSES.

The RTRACP requires the operator to size the process limits for the RTRACP before starting RTR on all platforms. No direct control of the process quotas of the RTRACP is offered for UNIX based platforms. However, log file entries will result if hard limits are less than the preferred values for the RTRACP.

The START RTR qualifiers /LINK and /PROCESSES apply only to the OpenVMS platform. Process quotas on UNIX platforms must be determined through operating system handling of virtual memory sizing.

In HP-UX and Linux clusters, to configure RTR in ACTIVE/STANDBY mode the following environmental variables must be set:

For more details on these environment variables, see Table 2.2, “RTR Environment Variables”.

Transaction data can be transmitted in a compressed format, to address the needs of users who wish to reduce their network bandwidth requirements for the transfer of such data. User data passed to RTR using the rtr_reply_to_client(), rtr_broadcast_event() and rtr_ext_broadcast_event() calls may optionally be compressed before being passed to the RTRACP process for transmission to the RTR router. The event or reply data are decompressed to their original state before being presented to the receiving applications.

Compression and decompression occur in the address space of client and server applications, so these processes will consume additional CPU resources when compression is in use. Compression is done using the zlib open source library.

To ensure interoperability, all participating systems must be running a version of RTR that supports compression. Failure to observe this restriction may cause compressed data to be incorrectly delivered to applications expecting uncompressed data.

You cannot use compression and the msgfmt argument of rtr_reply_to_client(), rtr_broadcast_event(), or rtr_ext_broadcast_event() simultaneously.

Compression is controlled with the following environment variables defined on participating frontends and backends as follows:

The EVENT_COMPRESS variables control event-data compression. They are deployed on a frontend to control compression of outgoing client broadcasts, and on a backend to control compression of outgoing server broadcasts. The LEVEL variable controls compression amount or level; valid values are integers in the range 1-9. Higher levels cause better compression, at the cost of increased CPU usage. If unspecified, the zlib default compression level is used.

The REPLY_COMPRESS variables control outgoing server reply data compression, and are consequently deployed on the backends where the replies originate.

The THRESHOLD variable defines the size of the largest data packet not subject to compression; anything larger is compressed. A threshold value of 0, the default, disables compression. These variables are read once only, shortly after an application makes its first call to the RTR API. No setup is required on router nodes, or if no compression is wanted. Decompression is automatic, provided the interoperability criteria (see above) are met.

RTR> show process/counter=api_compress*

RTR> show process/counter=*compressed

RTR> show process/counter=*event*bytes*
RTR> show process/counter=*reply_to_client_bytes*

RTR flow control manages message flow between processes and nodes controlled by RTR. Flow control regulates the sending rate of a process to achieve maximum message throughput and avoid resource exhaustion. RTR flow control is designed to deal with short-term system overload; if flow control indicates frequent resource depletion, the topology of the system should be evaluated. Use MONITOR TRAFFIC to check for congestion rates on RTR links between nodes.

Within a facility, RTR monitors four role directions:

Messages travel from a sender to a recipient at a rate, controlled by RTR, that the recipient can handle. RTR achieves this by limiting the rate of the sender using credits. Each sending RTRACP asks its partner for credit or permission to send. When a recipient is prepared to receive data, it grants credits. Each time a sender sends data, its credit is reduced until it is exhausted. Exhausted credit must be replenished before more data can be sent. Availability of credits can be checked with the MONITOR FLOW display. If an application is sending many large messages, flow control may not close the application channel soon enough. This may cause the RTRACP to run out of memory.

For applications using large messages containing tens of thousands of bytes, the following environment variables are provided to adjust the flow control algorithm parameters: RTR_MAX_CHANNEL_WAITQ_LIMIT, RTR_MAX_CHANNEL_WAITQ_BYTES, and RTR_MAX_CHANNEL_FULL_COUNT. These variables are described in Table 2.2, “RTR Environment Variables”.

When the WAITQ memory quota is exceeded, the RTRACP deletes all messages in the queue and returns a status of RTR_STS_CACHEXHAU, along with a "channel has been closed" message (rtr_mt_closed) which indicates that all messages were discarded in the application channel's queue.

Facilities do not require channels but networks links do. On OpenVMS, the number of channels is defined by the parameter CHANNELCNT that can be updated by changing MODPARAMS.DAT and running AUTOGEN. UNIX uses sockets, not channels.

RTR V2 TIDs are 2 longwords in length. In certain circumstances with multiple interface cards and no DECnet network, a V2 API may encounter a non-unique TID seen across nodes due to a non-unique value in the high-order 2 bytes of the 2nd longword. You can uniquely set the high-order 2 bytes of the 2nd longword with the RTR_TID_OPTION environment variable using one of two possible settings:

$ DEFINE/SYS RTR_TID_OPTION value   !where 0 < value < 65536
$ DEFINE/SYS RTR_TID_OPTION -1      !uses the SCSSYSTEMID

It is possible to override the protocol failover mechanism by specifying the transport protocol to be employed for a link by naming links to include a transport selecting prefix. Prefixing names of nodes with tcp. and dna. specifies TCP/IP or DECnet as the required transports respectively. These prefixes cause the local node to employ only the specified transport protocol when attempting a connection on the link to which the prefix has been applied. Using a protocol prefix on one node does not prevent a remote node from connecting using some other transport.

RTR> create facility FAX1 /frontend=(dna.FE1,dna.FE2,dna.FE3 ....)
/router=(dna.DNET1,dna.DNET2)
/backend=(dna.SRV1,dna.SRV2)

RTR> create facility FINANCE
/frontend=(tcp.client1,tcp.client2,tcp.client7 ....)
/router=(tcp.routr1,tcp.routr2)
/backend=(tcp.srv1,tcp.srv2)

When using RTR with Dynamic Host Configuration Protocol (DHCP) or an internet tunnel, a node name may not be fully known; special naming techniques are provided for these conditions.

Anonymous Clients

RTR allows the use of wildcards when specifying the frontends from which a router is permitted to accept connections (that is, in the facility definition on the router). Valid wild card characters are “*”, “%” and “?”. The result of using a wild card character at facility configuration time is the creation of a template link. definition on the router). Valid wildcard characters are asterisk (*), percent sign (%), and question mark (?). Using a wildcard character at facility configuration time will result in the creation of a template link.

When operating RTR in conjunction with internet tunnels, a client system outside of the corporate firewall uses tunnel software to obtain a secure channel from the Internet to inside the corporate domain. The tunnel client is assigned an address by the tunnel server from a pool when the tunnel software starts up.

RTR> create facility . . ./frontend=*.pool.places.dec.com

This command enables all nodes connecting through the tunnel to connect as frontends. The anonymous client feature may also be used with frontends that are using DHCP for TCP/IP address assignment.

Using the Tunnel Prefix

RTR> create facility . . ./router=router.rtr.dec.com -
/frontend=tunnel.client.rtr.dec.com

RTR> create facility /router=router.rtr.dec.com -
/frontend=client.rtr.dec.com

Troubleshooting Tunnel and Wildcard Connections

To assist in diagnosing connect-acceptance problems, use the monitor picture ACCFAIL. This picture displays the recent history of those links from which the local node has refused to accept connections. It also displays the failed link name as provided by the network transport, and can assist in rapidly identifying any problems.

TCP Services File

RTR uses the TCP/IP port number 46000 for the network communication daemon rtr rtrd. Refer to the VSI Reliable Transaction Router Installation Guide for how to reserve this port for UNIX systems.

TCP Port Usage

If usage of TCP is enabled, the RTR usage of TCP ports is as follows:

set RTR_PREF_PROT=RTR_TCP_FIRST
set RTR_PREF_PROT=RTR_TCP_ONLY
set RTR_PREF_PROT=RTR_DNA_FIRST
set RTR_PREF_PROT=RTR_DNA_ONLY

These statements set the choice of network transport to TCP/IP with fallback to DECnet, TCP/IP only, DECnet with fallback to TCP/IP or DECnet only.

Network Protocol Selection on OpenVMS

$ DEFINE/SYSTEM RTR_PREF_PROT RTR_DNA_FIRST

If you are using TCP/IP, you must use the node name prefix dna. if you want DECnet transport to be used.

If you are using DECnet as the default, you must use the node name prefix tcp. to connect to other nodes using TCP/IP transport.

If the value of the logical RTR_PREF_PROT is changed, the new value takes effect only after RTR has been restarted.

VSI Reliable Transaction Router Version 5.1 for OpenVMS can use either TCP/IP Services for OpenVMS or TCPware Version 5.1 as the TCP/IP transport layer. For more details on modifying RTR$STARTUP.COM on OpenVMS, see the full OpenVMS installation instructions in the VSI Reliable Transaction Router Installation Guide.

Restricting Network Connections

Ordinarily RTR accepts network connections on any configured interface in the system. Connections may optionally be limited to a specific interface by specifying the interface name with the environment variable RTR_BIND_HOST. This variable is read by the ACP, daemon and command server processes. The name supplied must be resolvable to an IP address (using gethostbyname( )). Note that only the first IP address obtained is used.

It is an error if the value of the bind host cannot be resolved, or translates to an interface that cannot be bound, and entries in the RTR log file will result. The affected process will continue to run, but will be unable to accept incoming network connections.

You may need to set up a dual-rail (multihome) environment to accommodate a firewall, segregate a network subnet or possibly to ease the load on an Ethernet line. In some situations, this can improve performance.

Physical Network Card Setup

For example, the configuration shown in Figure 2.3, “Dual-Rail Configuration with Network Cards on a Router” illustrates a firewall in a configuration with three RTR nodes and two network cards installed on the router.

In Figure 2.3, “Dual-Rail Configuration with Network Cards on a Router”, Node A is a frontend, Node B, with the two network cards (nc), is both a router and a backend, and Node C is a backend. The hubs are Ethernet hubs. Figure 2.4, “Dual-Rail Configuration with Network Cards on a Frontend” illustrates a frontend with two network cards.

RTR Facility Setup

RTR resolves addresses to one name in the DNS Server when you use a wildcard for frontends from a router.

A host with more than one network interface is multihomed. In a multihomed configuration, care must be taken to ensure that the gethostbyname function returns the list of all possible network addresses for the host. Otherwise, RTR may reject connections when it cannot recognize the host. To return the address list, use a correctly configured DNS. Using the /etc/hosts file on a UNIX server does not return the list of addresses.

Networking support for machines with multiple network adapters allows multiple IP connection targets for any host. With this capability, any pair of machines connected by multiple network paths can fail over to an alternate path if the primary path becomes unusable.

RTR determines the set of IP addresses to be used for a remote host when the host name is looked up using the gethostbyname() API. Depending on your platform and site policies, the IP address information will be provided by UNIX hosts file entries, OpenVMS TCPIP hosts entries, or by one or more BIND servers. Examples for a system named 'host1' with two interfaces follow:

UNIX hosts file:

1.2.3.4    host1_interfaceb
1.2.4.4    host1 host1_interfaceb

OpenVMS:

TCPIP> set host "host1_interfaceb"/address=1.2.3.4
TCPIP> set host "host1"/address=1.2.4.4/alias="host1_interfaceb"

Given the information above, RTR will attempt to connect to remote system 'host1' using address 1.2.4.4 first. Should this connection attempt fail, RTR will retry using address 1.2.3.4.

Connection attempts that invoked address failover can be monitored using the RTR monitor picture Netstat.

Note that connection attempts using IP to unreachable hosts usually terminate with a timeout condition, but are often intercepted by the RTR connection timeout whose default value is 60s. This is followed by a further quiescent period whose default value is 90s. You may wish to consider changing the values for these timers for a faster reconnection rate.

Linux INET Interface Support:

– SAMPLE /etc/hosts file –
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1        hostname localhost.localdomain   localhost
----------------------------

Thus, the gethostbyname() function returns address 127.0.0.1 for any query on hostname. RTR on Linux ensures smooth operation by enumerating all INET interfaces, adding to its cache the IP address associated with each INET interface that is not a loopback interface and is UP (as shown by the ifconfig command).

To view the interfaces on your system, issue the command ifconfig -a; this command reveals which interfaces are loopback, and which are UP.

If a tunnel separates the frontends from the routers, configure the frontends on the routers with names corresponding to the pseudo-adapter addresses assigned by the tunnel. If these are unpredictable, you can use wildcards on the routers only.

If a tunnel separates the routers and the backends, configure each with respect to the other with the name prefix "tunnel."

When starting RTR, the Service looks for the file UsrStart.RTR in the RTR home directory. When it finds the file, the Service executes any RTR commands it may contain. RTR commands from UsrStart.RTR execute after RTR has been started.

From the point of view of the Service, the RTR home directory is found in the system-level environment variable RTR_DIRECTORY, or, if that is not defined, then the directory from which the Service was executed.

For the RTR Service to use it, RTR_DIRECTORY must be defined in the system-level environment variables list, not the user-level environment variables list. Also, the system must be rebooted after the definition of RTR_DIRECTORY is either created or changed for it to be used.

If you put STOP RTR in the UsrStart.RTR file, it will stop RTR. The Service will not detect that RTR has been stopped and will offer only the STOP action button. Pressing the STOP button will fix the problem.

When the Service stops RTR, it recreates SrvcIn.Txt and creates RTRStop.RTR for stopdown commands. Creation of these files is unconditional; that is, they are created every time RTR is started or stopped, whether or not they already exist. RTR will therefore ignore (and overwrite) any changes made to one of these files.

Figure 2.5, “Assignment of Sequence Numbers in a Shadow Configuration” shows how sequence numbers are initially assigned in a simple partition with two backends named Bronze and Gold, and a router named Silver.

SET PARTITION test/PRIORITY_LIST=(bronze,gold)

Use the same priority list order on all partition members. If a different list is used, the router will determine the sequence number for conflicting members through the order in which those members joined the facility. For example, if the above command were issued only on Bronze, and Gold had the opposite priority list, the router would assign the lower sequence number to the backend that joined the facility first.

Use the SET PARTITION command with the /FAILOVER_POLICY qualifier to establish whether to fail over to a shadow or a standby backend. For example, use the /FAILOVER_POLICY qualifier to select a new active primary in configurations where shadowing is enabled. This qualifier takes precedence over the /PRIORITY_LIST qualifier. Use the /PRIORITY_LIST qualifier to determine the failover order for specific nodes. It is most useful in cluster configurations where it can specify the exact failover order for the nodes within the cluster.

For example, in a standby facility on a cluster of four nodes, the /PRIORITY_LIST qualifier can specify the desired order of failover for those cluster members. Some machines within a cluster may be more powerful than other machines. This feature allows for the most efficient use of those machines.

OpenVMS and Tru64 UNIX clusters use mechanisms such as a lock manager to deal with file sharing and sustained availability. The cluster configuration includes dual-ported disks to provide access from multiple CPUs.

HP-UX and Linux clusters, unlike OpenVMS and Tru64 Clusters, are not shared - all clusters. HP-UX uses HP Serviceguard and Cluster File System (CFS) and Red Hat Linux uses the Cluster-suite and Global File System (GFS) for clustering.

For enabling RTR to recognize the backend nodes of the cluster participating in ACTIVE/STANDBY configuration, the following environmental variable should be set:

RTR_CLU_MEM_NODES= "BE1, BE2"

In a ACTIVE/STANDBY configuration, if one of the nodes in the cluster fails, then RTR on the other node opens the failed node’s RTR journal and recovers any uncertain transactions from it, thereby ensuring transaction consistency.

Windows clusters, unlike OpenVMS and Tru64 Clusters, are not shared-all clusters. Windows clusters use the concept of host-based clustering, that is, one node physically mounts the shared disks and makes the shared disks available as a network share to all other nodes in the cluster. If the host node fails, then one of the other nodes will rehost the disks. This rehosting is handled by the Windows clustering software. Only two-node Windows cluster configurations are supported for RTR. In terms of Windows clusters, RTR is an application and the RTR journals are the database resource that fails over between the Windows cluster servers. (A good reference for Windows clustering information is Joseph M. Lamb's Windows 2000 Clustering and Load Balancing Handbook available from Prentice-Hall.)

Unrecognized or unsupported clusters have different behavior than recognized or supported clusters.

The default behavior in unrecognized cluster systems is to treat them as non-clustered. However, RTR standby failover will still work. RTR will fail over to the standby server process if the active server process fails. This standby takeover also performs recovery. If it is only the active server process that failed, then RTR can still recover any uncertain transactions through the remote RTR process. If, however, the node itself becomes unavailable (from, for example, an RTR crash, a node crash or a network crash) then the recovery process performs a journal scan to locate the journal of the failed node.

But unlike the case of a recognized cluster, RTR does not wait for the journal to become available. Instead, it changes to the active state and continues to process transactions. Any incomplete transactions in the failed node's journal will remain there; these transactions are not lost. They are eventually recovered when the failed node becomes active again, although their sequencing will be lost.

Partitions are subdivisions of a routing key range of values. They are used with a partitioned data model and RTR data-content routing. Partitions exist for each distinct range of values in the routing key for which a server is available to process transactions. RTR provides for failure tolerance by allowing system operators to start separate instances of partitions in a distributed network and by automatically managing the state and flow of transactions to the partition instances.

The system operator can issue commands to control certain partition characteristics, and to set preferences concerning partition behavior.

A valid partition name can contain no more than 63 characters. It can combine alphanumeric characters (abc123), the underscore (_), and the dollar sign ($). Partition names must be unique within a facility name and should be referenced on the command line with the facility name when using partition commands. Partition names exist only on the backend where the partition resides. You will not see the partition names at the RTR routers.

Partitions can receive automatically generated default names, in the form RTR$DEFAULT_PARTITION, unless the name is supplied.

Using this model, the partition segments and key ranges served by the server are still specified by the server when the channel is opened.

The system manager can supply partition names using the create partition system management command, or by using rtr_open_channel() flag arguments. The system manager can set partition characteristics with this command and applications can open channels to the partition by name. See Section 3.4, “Binding Server Channels to Named Partitions” for an example of passing a partition name with rtr_open_channel().

Partitions are created implicitly when an application program calls rtr_open_channel() to create a server channel, specifying the key segments and value ranges for the segments with the pkeyseg argument. Other partition attributes are established with the flags argument. Prior to RTR V3.2, this was the only way partitions could be created. Partitions created in this way are automatically deleted when the last server channel to the partition is closed.

Partitions can also be created by the system operator before server application program start up using system management commands. This gives the operator more control over partition characteristics. Partitions created in this way remain in the system until either explicitly deleted by the operator, or RTR is stopped.

The name of the facility in which the partition resides can be specified with the /FACILITY command line qualifier, or as a colon-separated prefix to the partition name (for example, Facility1:Partition1). Detailed descriptions of the command syntax are given in the Command Line Reference section of this manual, and are summarized in the following discussions. Examples in the following sections use a partition name of Partition1 in the facility name of Facility1.

The rtr_set_info() call completes asynchronously. If the function call is successful, completion is signaled by the delivery of an RTR message of type rtr_mt_closed on the channel whose identifier is returned through the pchannel argument. The programmer should retrieve this message by using rtr_receive_message(). The data accompanying the message is of type rtr_status_data_t. The completion status of the partition command can be accessed as the status field of the message data.

RTR> SET PARTITION/SHADOW Facility1:Partition1

For further information, see the SET PARTITION command in Chapter 9, RTR Commands.

rtr_qualifier_value_t    set_qualifiers[ 2 ];
rtr_partition_state_t    newState = rtr_partition_state_shadow;
set_qualifiers[ 0 ].qv_qualifier = rtr_partition_state;
set_qualifiers[ 0 ].qv_value = &newState;
set_qualifiers[ 1 ].qv_qualifier = rtr_qualifiers_end;
set_qualifiers[ 1 ].qv_value = NULL;

To disable shadowing, specify newState as rtr_partition_state_noshadow.

Transaction presentation is the process of passing transactions to idle server channels for processing. While transaction presentation is active, new transactions are started on the first free server channel for the appropriate partition.

Use the /SUSPEND qualifier to the SET PARTITION command to halt the presentation of new transactions to servers on the backend where the command is entered. The command completes when the processing of all currently active transactions is complete. The optional /TIMEOUT qualifier specifies, as a number of seconds, the time that the command waits for completion. If the command times out, presentation of new transactions are suspended, but there still exist transactions for which servers have yet to complete processing. The operator must decide either to reenter the command and wait a further period of time, or resume the partition. Note that use of this command does not affect any transaction timeout value specified by RTR clients, so such transactions may encounter a timeout condition if the partition remains suspended.

The /RESUME qualifier restarts presentation of transactions to the server application channels.

RTR> SET PARTITION/SUSPEND/TIMEOUT=5 Facility1:Partition1
RTR>
RTR> SET PARTITION/RESUME Facility1:Partition1

For a more complete description, see the SET PARTITION command in Chapter 6.

rtr_qualifier_value_t    set_qualifiers[ 3 ];
rtr_partition_state_t    newState = rtr_partition_state_suspend;
rtr_uns_32_t    ulTimeoutSecs = 30;
set_qualifiers[ 0 ].qv_qualifier = rtr_partition_state;
set_qualifiers[ 0 ].qv_value = &newState;
set_qualifiers[ 1 ].qv_qualifier = rtr_partition_cmd_timeout_secs;
set_qualifiers[ 1 ].qv_value = &ulTimeoutSecs;
set_qualifiers[ 2 ].qv_qualifier = rtr_qualifiers_end;
set_qualifiers[ 2 ].qv_value = NULL;

Note that the timeout is an optional element. To resume transaction presentation, specify newState as rtr_partition_state_resume.

The purpose of RTR automated recovery is to ensure the best possible consistency of application databases across a distributed computing environment. To achieve this, RTR relies in part on information stored in the journals of the participating backends. Should one or more of these systems be unavailable at recovery time, automated recovery may stall or fail awaiting availability of these systems and their journals. This enforces data consistency where transaction order is important, but can affect application availability.

For example, if a partition enters a wait state or fails, but has neither a local or remote journal, an operator can instruct RTR to skip the current step in the recovery process with the /IGNORE_RECOVERY qualifier. Since this command bypasses parts of the recovery cycle, use it with caution in cases where availability is valued over consistency in application databases. For a read-only database, transaction order will be unimportant.

The recovery cycle can also be manually restarted with the /RESTART_RECOVERY qualifier. This may be useful if the operator previously aborted automated recovery. Since this command can result in recovery of transactions from previously inaccessible journals, do not use this if your applications are sensitive to the order in which transactions are processed by the servers.

RTR> SET PARTITION/IGNORE_RECOVERY Facility1:Partition1
RTR>
RTR> SET PARTITION/RESTART_RECOVERY Facility1:Partition1

A complete description of the SET PARTITION command qualifiers can be found in Chapter 6.

rtr_qualifier_value_t    set_qualifiers[ 2 ];
rtr_partition_state_t    newState = rtr_partition_state_exitwait;
set_qualifiers[ 0 ].qv_qualifier = rtr_partition_state;
set_qualifiers[ 0 ].qv_value = &newState;
set_qualifiers[ 1 ].qv_qualifier = rtr_qualifiers_end;
set_qualifiers[ 1 ].qv_value = NULL;

To restart recovery, specify newState as rtr_partition_state_recover.

RTR> SET PARTITION/PRIORITY_LIST=(BE1, BE2, BE3) Facility1:Partition1

Suspend transaction presentation before entering or changing the priority list.

Chapter 9, RTR Commands provides more information on the SET PARTITION command.

rtr_qualifier_value_t     set_qualifiers[ 2 ];
char    *szNodeList = "your,list,of,node,names,here"
set_qualifiers[ 0 ].qv_qualifier = rtr_partition_be_priority_list;
set_qualifiers[ 0 ].qv_value = &szNodeList;
set_qualifiers[ 1 ].qv_qualifier = rtr_qualifiers_end;
set_qualifiers[ 1 ].qv_value = NULL;

The time required to effect a failover and the subsequent impact on client response times will influence choice of failover policy. The time for standby takeover of a failed node's journal depends on the size of that journal, though failover to a shadow site is affected quickly. However, if the secondary shadow site has accumulated a backlog of transactions, they must be processed before any new transactions can be started. The choice will be determined by the characteristics of your application and configuration.

 RTR> SET PARTITION/FAILOVER_POLICY=SHADOW Facility1:Partition1

RTR> SET PARTITION FacALPHA:AtoG/FAILOVER_POLICY=STAND_BY

You can view the policy that has been set with the SHOW PARTITION/FULL command.

For more information see the SET PARTITION command in Chapter 9, RTR Commands.

rtr_qualifier_value_t    set_qualifiers[ 2 ];
rtr_partition_failover_policy_t newPolicy;
set_qualifiers[ 0 ].qv_qualifier = rtr_partition_failover_policy;
set_qualifiers[ 0 ].qv_value = &newPolicy;
set_qualifiers[ 1 ].qv_qualifier = rtr_qualifiers_end;
set_qualifiers[ 1 ].qv_value = NULL;

After a failure, RTR replays transactions stored in its journal as appropriate. RTR has implemented the capability of controlling transaction replay in cases where a killer message happens during a transaction replay preventing recovery from continuing normally. A killer message causes server availability to be lost because of the presence of a message capable of causing repeated server application failure during recovery. This is typically the result of an improperly handled condition or application programming error within the server itself. Under such circumstances it may be desirable to sidestep a particular transaction, maintain server operation, and manually process the transaction at some later time.

The RTR solution is to establish, for a given partition with the SET PARTITION command, the maximum number of retries for any given transaction presented during recovery. Once this limit has been exceeded, the offending transaction is removed from the recovery process and is written to the journal as an exception record. Subsequent processing of this transaction requires manual intervention by someone qualified to evaluate and correct the situation in both the application and in RTR. Once the application status is understood, the SET TRANSACTION command can be used to update the journal, thus ensuring that the final state of any manually transacted exceptions are accurately reflected in future recovery operations.

The recovery retry count is partition-specific, and applies to both local and shadow recovery operations. The default is no limit on the number of retries, which permits a killer message to bring down all available servers servicing a given partition.

The recovery retry count should be set before starting (or restarting) the application servers so that the limit is established prior to the start of recovery operations.

 RTR> SET PARTITION/RECOVERY_RETRY_COUNT=3 Facility1:Partition1 

See Chapter 9, RTR Commands for more information on the SET PARTITION command.

rtr_qualifier_value_t     set_qualifiers[ 2 ];
rtr_uns_32_t    newLimit    = . . .;
set_qualifiers[ 0 ].qv_qualifier = rtr_partition_rcvy_retry_count;
set_qualifiers[ 0 ].qv_value = &newLimit;
set_qualifiers[ 1 ].qv_qualifier = rtr_qualifiers_end;
set_qualifiers[ 1 ].qv_value = NULL;

The retry limit applies to all transactions that have reached the voting stage on a server. If a server always dies before voting on a transaction, RTR aborts the transaction after the third try (“three-strikes and you're out!”). For more information, see the description of /RECOVERY_RETRY_COUNT for the SET PARTITION command.

Partitions in RTR are designed to be persistent, remaining until explicitly removed during normal RTR processing. However, under certain conditions, relics of partitions can remain in the RTR journal. RTR automatically performs some cleanup of such records, but depends on the creation of the relevant facility to initiate this process. For example, in a test environment, many facilities are created for temporary use, with no intention of retaining those facilities. Because the creation of each facility may cause the creation of associated records for a partition in the RTR journal, creating many ad hoc facilities can cause the RTR journal to become filled. In such a case, when trying to create a new partition (or opening a new server channel), the error message NOMOREPRT may appear.

To correct this problem, the journal must be purged of these ad hoc entries. To purge the RTR journal of such unwanted transactions, use the DUMP JOURNAL command to verify the partition name and transaction ID of the unwanted transactions, and use the SET TRANSACTION command with the partition name and transaction ID to set the state to DONE. Recreate the facility with the CREATE FACILITY command.

EXCEPTION transactions keep the application available, although they cause some loss of data integrity. EXCEPTION transactions are considered committed by the initiator of the transaction, as well as by the other participants (such as the other shadow member). Therefore, subsequent transactions, which depend on the results of this transaction, could produce erroneous outcomes. In some applications, the erroneous outcomes do not matter, but in applications where the outcome does matter, the best approach is to allow the system administrator to manually intervene.

The shadow events are delivered with no special status and no data. They are delivered only to the servers whose state has changed.

A server receives RTR_EVTNUM_SRSTANDBY when it starts up and servers already exist for the same key range on another node in the same cluster.

A server receives RTR_EVTNUM_SRSECONDARY when it starts up and a shadow primary set of servers exist elsewhere.

A server receives RTR_EVTNUM_SRSHADOWLOST if it is running as primary and the secondary goes away.

A server receives RTR_EVTNUM_SRSHADOWGAIN if it is running as primary and a secondary node starts up.

A server receives RTR_EVTNUM_SRRECOVERCMPL when it has finished doing recovery operations and is ready to start processing new transactions.

RTR need only be installed; it can be on a Windows node defined with any RTR role. The counter host can also be on an HP machine running HP OpenView, which can handle Windows performance tools.

A target machine is any RTR node in your network which you wish to include in your RTR performance counter coverage. Coverage is configured by including the name of the target machine on the command line used to start the counter host. A target machine must accept RTR remote commands from any counter host specifying the machine as a target. This requires TCP connectivity and permission for the user on the counter host to operate RTR and use rsh for remote access on all target machines. Target machines may be of any RTR platform type, that is, not just Windows.

To generate a web page allowing access to the performance counters for a system, the Windows system hosting the counters must contact the RTR HTTP web server machine with registration information. This registration is automatic, and occurs periodically. The message “no counter host for target ...” indicates that no such registration can be found on this web server for the indicated target machine.

The most likely reason for this condition is that no performance counter host has been configured to provide coverage for the target in question. For information on starting up counter hosting, see Section 6.3, “Starting and Stopping Performance Counter Hosting”.

This message may also be observed if the RTR HTTP server has only recently been started and asked to display the performance chart page before receiving the necessary registration information. Since counter registration occurs automatically and periodically, just reload the page later.

Performance counter hosts register their activity with covered target machines so that RTR management web pages may be generated to display the performance counter charting application. A warning appears if this registration data is more than 10 minutes old. Check that the performance counter host is functioning correctly.

The data in the Windows performance objects is periodically refreshed as new samples are taken from the target systems. The rate at which this information is updated depends on both the value of interval_ms specified when configuring the performance counter host, and the latency of the network connecting the host and target. Latency can be minimized by placing hosts near the targets for which they provide coverage.

The default setting of the Windows performance tool charting function is to update the screen display every second. For performance counters that display as a rate of change per second, this will lead to erratic estimates of the rate in the displayed data, including false indications of zero rate, due to the mismatch in display and data update rates. VSI recommends that the display update rate be set to no less than 10 times the data sampling rate. For example, if the performance counters display per second, set the screen update interval at 10 seconds.

CPU usage on the counter host is inversely related to the value of interval_ms.

Where possible, the instance names of the Windows performance counters are the same as the underlying RTR objects being measured. However, not all names allowed by RTR are valid Windows counter instance names. Names containing the '$' character are modified by replacing that character with a hyphen for use in Windows.

Certain difficulties can be more easily investigated if a snapshot of the problem node is made. Make a snapshot if the application hangs, causes delays, or seems to be causing other problems.

OpenVMS

$  @SYS$MANAGER:RTR$SNAPSHOT.COM

The output is a file named nodename_RTR_DIAGS.TMP.

Information in this file can help to determine the possible causes of a fault (OpenVMS, DECnet, RTR, environment, database, application, and so on.) The information includes numerous RTR monitor pictures, executable image versions, process states, and so on.

UNIX

 > rtr_snapshot.sh 

The information displayed on the screen includes many RTR monitor pictures, executable image versions, process states, and other information.

Windows

To take a snapshot on Windows, click on the Snapshot icon on the RTR menu. A DOS-style window with the title ‘‘Snapshot’’ appears as the snapshot is taken. The file rtr_snapshot.log is created in the directory where RTR runs, for example, C:\Program Files\RTR . You can read this file with an editor such as Notepad.

OpenVMS

Certain potential difficulties can be more easily investigated by RTR Support if a dump of the RTR ACP is available. It shows diagnostics generated if unhandled exceptions occur.

The file SYS$MANAGER:RTR$STARTUP.COM can be altered to include the definition of the logical name RTR$DUMP_DIRECTORY which specifies the device and directory where the dump is to be generated.

Since an RTR dump file typically uses about 5000 blocks, enough space should be available on the chosen disk. For a very large node installation, or a large number of links, the dump file may be up to 20,000 blocks.

$ RTR
RTR> SET MODE /UNSUPPORTED
RTR> DEBUG ACP
RTR> SET MODE /NOUNSUPPORTED
$

Unsupported commands should be used with care.

UNIX Systems

UNIX core files are generated with no special configuration, but their name and location may vary depending on operating system settings and how RTR is started up.

Windows systems

(%WINDIR%\drwtsn32 -i)

The files created are %WINDIR%\DRWTSN32.LOG and %WINDIR%\USER.DMP.

These files should be included with any problem report submitted to RTR Engineering in the event of an RTR crash, along with the RTR dump file (RTR_<n>.DMP) and the RTR log file.

To provide the RTR Error Log, run RTR with logging in use. This creates the RTR Error Log, rtr_error*.log. This file resides in the RTR$DUMP_ DIRECTORY, as listed by operating system in the table below.

Displays counts of messages and number of bytes from RTRACP to the application, as viewed from a specific node. Includes opened, closed, msg1, msg1_uncertain, msgn, reply (reply to client), rettosend (return to sender), accepted, prepared, rejected, user_event, rtr_event, prepare, request_info, and set_info messages as appropriate. For receive_message and user_wakeup, displays calls, active, fail, and timeout counts. Refer to the VSI Reliable Transaction Router C Application Programmer’s Reference Manual for an explanation of the message types.

RTR ACP to Application Messages, Node: NodeA  PID: 1568  Process name: -ALL-
    Image: -ALL-                                  14:15:46 Tue Dec 04 2001
Message Type        Client            Server             Other        Pend
                    #      Bytes      #     Bytes        #    Bytes     #
opened              4        760      3        38        0        0     8
closed              0          0      1       136    43492   295745     1
msg1                                  0       170                       2
msg1_uncertain                        0         0                       0
msgn                                  0       253                       3
reply               0          0      0         0                       0
rettosend           0          0                                        0
prepared            0          0                                        0
accepted            0          0      0         0                       0
rejected            0          0      0        68                       1
user_event          0          0      0         0                       0
rtr_event           0          0      0         0                       0
prepare                                                  0        0     0
                    Other
request_info                                         43347  5211764     0
set_info                                                 0        0     0
                  Calls   Active     Fail   Timeout
receive_message   88607        1     1759      1759
user_wakeup           0        0

Displays a list of RTR processes, and for each process the number of transactions they have started, the number of transactions they have completed, and the number of transactions still active.

ACTIVE TRANSACTIONS BY PROCESS  Fri Mar 12 1999 19:32:41

                                              Starts  Complete  Active
Display totals:                                    5         5       0

Node      ID      Process name   Image name
NodeA     11141   SMITH_1        DISK01:[SMITH]RTR 5         5       0

Displays counts of RTR alarms. Shows the kilobytes sent, kilobytes received, number of alarms sent, and number of alarms received for internal alarm detection on the target node. Each node on the RTR network detects its own alarms, and then sends the alarms to their associated router. For backends, that means all routers. For routers, it means itself. For frontends, it means the router to which it is connected.

Thus each node may send alarms, and routers may receive alarms.

            Alarm Detection Traffic at 11:28:46 Wed May 28 2003
                                node: rage
Facility         KBytesSent     KBytesRcvd     AlarmsSent     AlarmsRcvd
Facility1              5.57           5.57             42             42
Facility2              3.29           3.29             10             10
Facility3              5.64           5.64             43             43
Facility4              3.36           3.36             11             11

Displays counts of messages and number of bytes from the application to RTRACP, as viewed from a specific node. Includes open_channel, close_channel, prepare_tx (prepare transaction), accept_tx (accept transaction), reject_tx (reject transaction), broadcast_event, start_tx, send_to_server, reply_to_client, request_info, and set_info. Refer to the VSI Reliable Transaction Router C Application Programmer’s Reference Manual for an explanation of the message types.

RTR Application to ACP Messages, Node: NodeA  PID:  1568 Process name: -ALL-
   Image: -ALL-                                  14:21:19 Tue Dec 04 2001
RTR Message Type     Client           Server               Other
                     #     Bytes      #     Bytes          #      Bytes
open_channel        11      8228      7      5376          0          0
close_channel        5       120      3        72          0          0
prepare_tx           0         0
accept_tx            0         0      0         0
reject_tx            0         0      0         0
broadcast_event      1        81      0         0
start_tx             2         0
send_to_server       5       548
reply_to_client                       0         0
request_info                                           43768   16015270
set_info                                                   0          0

Displays the channels opened by the CALL RTR_OPEN_CHANNEL command.

RTR CHANNELS BY TYPE PER PROCESS  16:41:13 Tue Dec 04 2001

                                                         Call-out
Node     ID  Process name  Image name  Client  Server  Router  Backend
Nodea  2162  SMITH_1       DISK01:[SMI... 6       3         0        0

Displays compression configuration and operation information. Information displayed includes the following:

COMPRESSION OVERVIEW BY PROCESS 17:38:13 Wed Oct 15 2003 on -ALL-

                         +-Replies-----------+ +-Events------------------+
                                                              ClientServer
Node, Process ID & name  Threshold Level Ratio Threshold Level Ratio Ratio
nodeaa 10712  10712              0     0             0     0
rtrm08 1940   usertest v3t_sr    1    -1 100.0       0    -1
rtrm08 2108   usertest v3t_sr    1    -1 100.0       0    -1
rtrm08 2072   usertest v3t_sr    1    -1 100.0       0    -1
rtrm08 2020   usertest v3t_sr  999    -1 100.0       0    -1
rtrm08 1856   usertest v3t_sr    0    -1             0    -1
rtrm08 2120   usertest v3t_sr    0     9             0    -1
rtrm08 2112   usertest v3t_sr    0     9           555    -1 100.0 100.0
rtrm08 156    usertest v3t_sr    0     9           555     3 100.0 100.0
rtrm08 2168   usertest rtr.ex  777     9  91.0     666     3  88.4  88.1

Displays event routing data by facility, with totals. Destination columns count the number of events received by the local node and facility delivered to application channels, based on their subscriptions. Transit columns count the number of events that are in transit from the router to remote nodes in the destination facility.

The count columns contain the following:

    EVENT ROUTING STATISTICS BY FACILITY  Fri Sep 08 2000 15:21:47

                            Destination           Transit
Node & Facility           In    Out    Lost     In    Out   Lost
Total                    180    175       5    180    180      0

NODEA   FACCMS            25     25       0     25     25      0
NODEA   TESTFAC          155    150       5    155    155      0

Displays counts of RTR Explorer events. Shows the kilobytes sent, kilobytes received, number of alarms sent, and number of alarms received for backend data gathering on the target node. When RTR Explorer information is requested on a backend, either directly by using the GCS information class with API call rtr_request_info( ), or indirectly by viewing the RTR Explorer via a web browser, the backend must gather all configuration and status information from all routers.

Thus the routers may send this information, and the backends may gather this information.

            RTR Explorer Traffic at 11:34:53 Wed May 28 2003
                               node: rage
Facility   KBytesSent    KBytesRcvd   AlarmsSent   AlarmsRcvd
Facility1           4.47       4.47           20           20
Facility2           2.71       2.71            5            5
Facility3           4.54       4.54           21           21
Facility4           2.78       2.78            6            6

Displays information about journal usage, including total number of entries and records written, number of records read, and how many bytes were involved. Bar graphs showing current usage of journal blocks (as a percentage of the total) are also provided. The bar graphs appear under the first line of the display JNL_xxx_BLOCKS_IN_USE.

   JOURNAL USAGE ON NODE NODEA       AT 10:36:05 Tue Apr  6 1999

   LOCAL JOURNAL                      STANDBY JOURNAL(S)

   JNL_LCL_BLOCKS_IN_USE       128    JNL_RMT_BLOCKS_IN_USE        0
   [___             13%          ]    [                0%          ]
   JNL_LCL_NR_BLOCKS           992    JNL_RMT_NR_BLOCKS          992
   JNL_LCL_TOP_BLOCKS_USED     128    JNL_RMT_TOP_BLOCKS_USED      0
   JNL_LCL_BLOCKS_AVAILABLE    864    JNL_RMT_BLOCKS_AVAILABLE     0
   JNL_LCL_TX_ENTRIES            1    JNL_RMT_TX_ENTRIES           0
   JNL_LCL_TX_RECORDS            2    JNL_RMT_TX_RECORDS           0
   JNL_LCL_MEMORY_BYTES     530121    JNL_RMT_MEMORY_BYTES      4197
   JNL_LCL_DISK_READS           31    JNL_RMT_DISK_READS          33
   JNL_LCL_BLOCKS_READ         992    JNL_RMT_BLOCKS_READ       1056
   JNL_LCL_DISK_WRITES          12    JNL_RMT_DISK_WRITES          0
   JNL_LCL_BLOCKS_WRITTEN       14    JNL_RMT_BLOCKS_WRITTEN       0
   JNL_LCL_ENTRIES_TOTAL       201    JNL_RMT_ENTRIES_TOTAL        5
   JNL_LCL_RECORDS_TOTAL       398    JNL_RMT_RECORDS_TOTAL        9
   JNL_LCL_RECORDS_READ         21    JNL_RMT_RECORDS_READ         0
   JNL_LCL_REC_BYTES_READ     8006    JNL_RMT_REC_BYTES_READ       0
   JNL_LCL_NONTX_ENTRIES         5    JNL_RMT_NONTX_ENTRIES        0

   JNL_LCL_OPEN_JOURNALS    2

This screen summarizes key areas managed by RTR showing, for example, the number of times RTR saved the day. All the counters describe the management by RTR as a whole, as opposed to per facility or per partition. The counters remain for the life of the RTRACP process. The table below describes the content of the Monitor Management rows.

              RTR Management at 15:23:09 Tue Jul 8 2003
                                  Node: NODEA

                          Backend Role Router Role
                          ------------ ------------
Restart Recoveries......          543           NA
Shadow Recoveries.......        19735           NA
Replay Recoveries.......           56           NA
Three-strikes Txns......            4           NA
Transactions Processed..      7385660      3958495
Partition Activations...          512           25
Link Establishments.....           57         5001

Shows the progress of transaction recovery. Last recovery backend is the last backend accessed to recover transactions. If the server state is lcl_rec_fail or shd_rec_fail, the Last Recovery Backend is the backend that could not be accessed. Journal scans is the number of journal files searched. Transactions recovered is the number of transactions found for this partition.

RECOVERY INFORMATION at Tue Apr  6 1999 10:54:50, on NODEA

                      Last      Restart-Recovery    Shadow-Recovery
              Server  Recovery  Journal  Txns       Journal  Txns
Partition     State   Backend   Scans    Recovered  Scans    Recovered
------------  ------  -------   -------  ---------  -------  ---------
RTR_PARTITA   active  NODEA           1          0        0          0
RTR_PARTITB   active  NODEA           1          0        0          0
RTR_PARTITC   active  NODEA           1          0        0          0

         System Status at 10:27:51 Fri Apr  9 1999
                           node: NODEA

     Resource                OK   Warning
Facility QUORUM states......          x
JOURNAL free space..........  x
Link CONNECTS...............          x
Link traffic STALLS.........  x
FLOW control credits........  x
PARTITION states............  x
CALL Msg outstanding..........        x
Transaction QUEUES............ x
Transaction REJECTS........... x
Broadcast EVENT discards...... x

   For additional detail about a resource,
   monitor the appropriate subsystem. To
   customize threshold values, edit
   SYSTEM.MON. If viewing this page in a
   browser, follow the hyperlinks above.

To customize threshold values, edit the file SYSTEM.MON.

Displays the state of the ACP channel wait queues by facility.

     WAITQ STATISTICS BY FACILITY Mon Jul 26 2004 14:09:46 mpeso -ALL
                Servers                                 Clients
    Current     Maximum       Timestamp     Current     Maximum     Timestamp
     451288      451288   26-Jul-2004 14:09       0           0
                        Servers              Clients
Node   Facility     Current  Maximum     Current  Maximum
mpeso        f1      451288   451288           0        0

Displays the state of the ACP channel wait queues by process.

WAITQ STATISTICS BY PROCESS Mon Jul 26 2004 14:10:04 mpeso ID: -ALL
                Servers                                 Clients
    Current     Maximum       Timestamp     Current     Maximum     Timestamp
     451288      451288   26-Jul-2004 14:09       0           0
                        Servers              Clients
Node   Proc. Id Process Name     Current  Maximum     Current  Maximum
mpeso  20409E6A GAISFORD_126      451288   451288           0        0