The C++ API is an object-oriented application programming interface for RTR. With the C++ API, as an application developer, you can design and implement applications to take advantage of object-oriented (OO) design principles, including patterns and code reuse. As Figure 1.1, “RTR Layers” illustrates, the C++ API sits on top of the C API, which is a layer on top of RTR.

Client applications and server applications can use both the C++ API and the C API or just the C++ API. The C++ API maps RTR data elements, data structures, and C API functionality into the object-oriented design world.

The C++ API:

C++ API and RTR Technology

The C++ API provides an object-oriented framework through which you can implement fault-tolerant systems. C++ API code resides beneath application or business logic code. Thus, the C++ API interfaces directly with RTR while application code does not. This transparency simplifies the development and implementation of transaction processing solutions with RTR.

OO Benefits

With the C++ API, applications can inherit functionality from RTR.

The C++ API Provides Ease of Use

C++ API Design

The C++ API upgrades RTR technology by providing a set of classes that streamlines the development and implementation of RTR transaction processing applications. The C++ API has been designed to:

Use of the C++ API

The C++ API can be used to:

The C API enables applications to be written with RTR calls using the C programming language. The C API is described in the VSI Reliable Transaction Router C Application Programmer’s Reference Manual.

A concurrent server is an additional instance of a server application running on the same node as the original instance. If one concurrent server fails, the transaction in process is replayed to another server in the concurrent pool.

The main reason for using concurrent servers is to increase throughput by processing transactions in parallel, or to exploit Symmetric Multiprocessing (SMP) systems. The main constraint in using concurrent servers is the limit of available resources on the machine where the concurrent servers run. Concurrent servers deal with the same database partition. They may be implemented as multiple channels in a single process or as channels in separate processes. For an illustration of concurrent servers, refer to the VSI Reliable Transaction Router Getting Started manual. By default, servers are declared concurrent.

When a concurrent server fails, the server application can fail over to another running concurrent server, if one exists. Concurrent servers are useful to improve throughput, to make process failover possible, and to help minimize timeout problems in certain server applications. For more information on this topic, see the section Section 3.2.6: Server-Side Transaction Timeouts later in this document.

Concurrent servers must not access the same file or the same record in a database table. This can cause contention for a single resource, with potential for performance bottleneck. The resources that you can usefully share include the database and access to global (shared) memory. However, an application may need to take out locks on global/shared memory; this would need to be taken into account during design. With some applications, it may be possible to reduce operating system overhead by using multiple channels in a process.

Performance in a system is usually limited by the hardware in the configuration. Evaluating hardware constraints is described in the VSI Reliable Transaction Router System Manager’s Manual. Given unlimited system resources, adding concurrency will usually improve performance. Before putting a new application or system built with RTR into production, the prudent approach is to test performance. You can then make adjustments to optimize performance. Do not design a system that immediately uses all the resources in the configuration because there will be no room for growth.

Using concurrency also improves reliability, because RTR provides server process failover (the “three strikes and you're out ” rule) when you have concurrent servers.

An application designer may decide to use threads to have an application perform other tasks while waiting for RTR, for example, to process terminal input while waiting for RTR to commit a transaction or send a broadcast.

To use multiple threads, you write your application as a threaded application and use the shared thread library for the operating system on which your application runs. Use one channel per thread (with the C API), or one TransactionController per thread (with the C++ API). The application must manage the multiple processes.

To use multiple channels in a thread, use the polled receive method, polling for rtr_receive_message (C API) or Receive (C++ API). The application must contain code to handle the multiple channels or transaction controllers. This is by far the most complex solution and should only be used if it is not possible to use multiple threads or concurrent processes.

When using multiple threads in a process, the application must do the scheduling. One design using threads is to allocate a single channel for each thread. An alternative is to use multiple channels, each with a single thread. In this design, there are no synchronization issues, but the application must deal with different transactions on each thread.

Using multiple threads, design and processing is more streamlined. Within each thread, the application deals with only a single transaction at a time, but the application must deal with issues of access to common variables. It is often necessary to use mutexes (resource semaphores) and locks between resources.

To use multiple partitions in your application, your database must be designed with multiple partitions. You may also be able to use multiple partitions when you have multiple databases. In general, using multiple partitions can reduce resource contention and improve performance. In the case where contention for database resources is causing performance degradation, partitioning or repartitioning your database can improve performance. To reduce resource contention in a database:

When you have multiple databases to which transactions are posted, you can also design your RTR application to use partitions and thereby achieve better performance than without partitioning.

RTR deals with router failures automatically and transparently to the application. In the event of a router failure, neither client nor server applications need to do anything, and do not see an interruption in service. Consider router configuration when defining your RTR facility to minimize the impact of failure of the node where a router resides. If possible, place your routers on independent nodes, not on either the frontend or backend nodes of your configuration. If you do not have enough nodes to place routers on separate machines, configure routers with backends. This assumes a typical situation with many client applications on multiple frontends connecting to a few routers. For tradeoffs, see the sections on Section 2.5, “Design for Performance” and Section 2.6, “Configuration for Operability” in this chapter.

Provide multiple routers for redundancy. For configurations with a large number of frontends, the failure of a router causes many frontends to seek an alternate router. Therefore, configure sufficient routers to handle reconnection activity. When you configure multiple routers, one becomes the current router. If it fails, RTR automatically fails over to another.

For read-only applications, routers can be effective for establishing multiple sites for failover without using shadowing. To achieve this, define multiple, nonoverlapping facilities with the same facility name in your network. Then provide client applications in the facility with the list of routers. When the router for the active facility fails, client applications are automatically connected to an alternate site. Read-only transactions can alternatively be handled by a second partition running on a standby server. This can help reduce network traffic.

When a router fails, in-progress transactions are routed to another router if one is available in that facility.

Failover of any server is either event-driven or timer-based. For example, server loss due to process failure is event-driven and routinely handled by RTR. Server loss due to network link failure is timer-based, with timeout set by the SET LINK/INACTIVITY timer (default: 60 seconds). For more information on setting the inactivity timer, see the SET LINK command in the VSI Reliable Transaction Router System Manager’s Manual.

For example, below illustrates the use of concurrent servers to process transactions for Partition A. When one of the concurrent servers cannot service transactions going to partition A, another concurrent server (shown by the dashed line) processes the transaction. Failover to the concurrent server is transparent to the application and the user.

2.3.2.2. Standby Servers

Standby servers provide additional availability and node-failure tolerance. Figure 2.2, “Transaction Flow on Standby Servers” illustrates the processing of transactions for two partitions using standby servers.

When the configuration is operating normally, the primary servers send transactions to the designated partition (solid lines); transactions “A” proceed through primary server A to database partition A and transactions “B” proceed through primary server B to database partition B. However, when the primary server fails, the router reroutes transactions “A” through the standby server A ^' to partition A, and transactions “B” through the standby server B ^' to database partition B. Note that standby servers for different partitions can be on different nodes to improve throughput and availability. For example, the bottom node could be the primary server for partition B, with the top node the standby. The normal route is shown with a solid line, the standby route with a dashed line.

When the primary path for transactions intended for a specific partition fails, the standby server can still process the transactions. Standby servers automatically take over from the primary server if it fails, transparently to the application. Standby servers recover all in-progress transactions and replay them to complete the transactions.

As shown in Figure 2.2, “Transaction Flow on Standby Servers”, there can be multiple standby servers for a partition.

2.3.2.2.1. Standby Support in a Cluster

Failover and transaction recovery behave differently depending on whether server nodes are in a cluster configuration. Not all “cluster” systems are recognized by RTR; RTR recognizes only the more advanced or “true” cluster systems. Figure 2.3, “RTR Standby Servers” shows the components that form an RTR standby server configuration.

Two nodes, N1 and N2, in a cluster configuration are connected to shared disks D1, D2 and D3. Disks D1 and D2 are dedicated to the RTR journals for nodes N1 and N2 respectively, and D3 is the disk that hosts the clustered database. This is a partitioned database with two partitions, P1 and P2.

Under normal operating conditions, the RTR active server processes for each partition, P1 _A and P2 _A run on nodes N1 and N2 respectively. The standby server processes for each partition run on the other node, that is, P1 _S runs on N2 and P2 _S runs on N1. In this way, both nodes in the cluster actively participate in the transactions and at the same time provide redundancy for each other. In case of failure of a node, say N1, the standby server P1 _S is activated by RTR and becomes P1 _A and continues processing transactions without any loss of service or loss of in-flight transactions. Both the active and standby servers have access to the same database and therefore both can process the same transactions.

Failover between RTR standby servers behaves differently depending on the type of cluster where RTR is running. Actual configuration and behavior of each type of cluster depends on the operating system and the physical devices deployed. For RTR, configurations are either true clusters or host-based clusters.

In this section the various failure situations are analyzed in more detail. This can help system managers to configure the system in an optimal way.

A transactional shadow server handles the same transactions as the primary server, and maintains an identical copy of the database on the shadow. Both the primary and the shadow server receive every transaction for their key range or partition. If the primary server fails, the shadow server continues to operate and completes the transaction. This helps to protect transactions against site failure.

To understand and plan for smooth inter-node communication you must understand quorum.

Quorum is used by RTR to ensure facility consistency and deal with potential network partitioning. A facility achieves quorum if the right number of routers and backends in a facility (referred to in RTR as the quorum threshold), usually a majority, are active and connected.

In an OpenVMS cluster, for example, nodes communicate with each other to ensure that they have quorum, which is used to determine the state of the cluster; for cluster nodes to achieve quorum, a majority of possible voting member nodes must be available. In an OpenVMS cluster, quorum is node based. In the RTR environment, quorum is role based and facility specific. Nodes/roles in a facility that has quorum are quorate; a node that cannot participate in transactions becomes inquorate.

RTR computes a quorum threshold based on the distributed view of connected roles. The minimum value can be two. Thus a minimum of one router and one backend is required to achieve quorum. If the computed value of quorum is less than two, quorum cannot be achieved. In exceptional circumstances, the system manager can reset the quorum threshold below its computed value to continue operations, even when only a minimum number of nodes, less than a majority, is available. Note, however, that RTR uses other heuristics, not based on simple computation of available roles, to determine quorum viability. For instance, if a missing (but configured) backend's journal is accessible, that journal is used to count for the missing backend.

A facility without quorum cannot complete transactions. Only a facility that has quorum, whose nodes/roles are quorate can complete transactions. A node/role that becomes inquorate cannot participate in transactions.

Your facility definition also has an impact on the quorum negotiation undertaken for each transaction. To ensure that your configuration can survive a variety of failure scenarios (for example, loss of one or several nodes), you may need to define a node that does not process transactions. The sole use of this node in your RTR facility is to make quorum negotiation possible, even when you are left with only two nodes in your configuration. This quorum node prevents a network partition from occurring, which could cause major update synchronization problems.

Note that executing the CREATE FACILITY command or its programmatic equivalent does not immediately establish all links in the facility, which can take some time and depends on your physical configuration. Therefore, do not use a design that relies on all links being established immediately.

If your configuration is reduced to two server nodes out of a larger population, or if you are limited to two servers only, you may need to make some adjustments in how to manage quorum to ensure that transactions are processed. Use a quorum node as a tie breaker to ensure achieving quorum.

For example, with a five-node configuration ( Figure 2.4, “Configuration with Quorum Node”) in which one node acts as a quorum node, processing still continues even if one entire site fails (only two nodes left). When an RTR configuration is reduced to two nodes, the system manager can manually override the calculated quorum threshold. For details on this practice, see the VSI Reliable Transaction Router System Manager’s Manual.

Frequently with RTR, you will partition your database.

Partitioning your database means dividing your database into smaller databases to distribute the smaller databases across several disk drives. The advantage of partitioning is improved performance because records on different disk drives can be updated independently - resource contention for the data on a single disk drive is reduced. With RTR, you can design your application to access data records based on specific key ranges. When you place the data for those key ranges on different disk drives, you have partitioned your database. How you establish the partitioning of your database depends on the database and operating systems you are using. To determine how to partition your database, see the documentation for your database system.

In some applications that use RTR with shadowing, the implications of transaction serialization need to be considered.

The secondary shadowed database needs to commit these transactions in the same order. RTR ensures that this happens, transparently to the application.

However, if the application cannot take advantage of partitioning, there can be situations where delays occur while the application waits, say, for transaction 2 to be committed on the secondary. The best way to minimize this type of serialization delay is to use a partitioned database. However, because transaction serialization is not guaranteed across partitions, to achieve strict serialization where every transaction accepts in the same order on the primary and on the shadow, the application must use a single partition.

Not every application requires strict serialization, but some do. For example, if you are moving $20, say, from your savings to your checking account before withdrawing $20 from your checking account, you will want to be sure that the $20 is first moved from savings to checking before making your withdrawal. Otherwise you will not be able to complete the withdrawal, or perhaps, depending upon the policies of your bank, you may get a surcharge for withdrawing beyond your account balance. Or a banking application may require that checks drawn be debited first on a given day, before deposits. These represent dependent transactions, where you design the application to execute the transactions in a specific order.

If your application deals only with independent transactions, however, serialization will probably not be important. For example, an application that tracks payment of bills for a company would consider that the bill for each customer is independent of the bill for any other customer. The bill-tracking application could record bill payments received in any order. These would be independent transactions. An application that can ignore serialization will be simpler than one that must include logic to handle serialization and make corrections to transactions after a server failure.

In addition to dependent transactions that can make serialization more complex, if the application uses batch processing or concurrent servers, ensuring strict serialization may be difficult.

In a transactional shadow configuration using the same facility, the same partition, and the same key-range, RTR ensures that data in both databases are correctly serialized, provided that the application follows a few rules. For a description of these rules, see Chapter 3, Implementing an Application , later in this document. The shadow application runs on the backends, processes transactions based on the business and database logic required, and hands off transactions to the database engine that updates the database. The application can take advantage of multiple CPUs on the backends.

2.4.4.1. Transaction Serialization Detail

Transactions are serialized by committing them in chronological order within a partition. Do not share data records between partitions because they cannot be serialized correctly on the shadow site.

Dependent transactions operate on the same record and must be executed in the same order on the primary and the secondary servers. Independent transactions do not update the same data records and can be processed in any order.

RTR relies on database locking during its accept phase to determine if transactions executing on concurrent servers within a partition are dependent. A server that holds a lock on a data record during its vote call (AcceptTransaction for the C++ API or rtr_accept_tx for the C API) blocks other servers from updating the same record. Therefore only independent transactions can vote at the same time.

RTR tracks time in cycles using windows; a vote window is the time between the close of one commit cycle and the start of the next commit cycle.

2.4.4.1.1. RTR Commit Group

RTR commit grouping enables independent transactions to be scheduled together on the shadow secondary. A group of transactions that execute an AcceptTransaction (or rtr_accept_tx call for the C API) call within a vote window form an RTR commit group, identified by a unique commit sequence number (CSN). For example, given a router (TR), backend (BE), and database (DB), each transaction sent by the backend to the database server is represented by a vote. When the database receives each vote, it locks the database and responds as voted. The backend responds to the router in a time interval called the vote window, during which all votes that have locked the database receive the same commit sequence number. This is illustrated in Figure 2.5, “Commit Sequence Number”.

To improve performance on the secondary server, RTR lets this commit group of transactions execute in any order on the secondary.

RTR reuses the current CSN if it determines that the current transaction is independent of previous transactions. This way, transactions can be sent to the shadow in a bunch.

In a little more detail, RTR assumes that transactions within the vote window are independent. For example, given a router and a backend processing transactions, as shown in Figure 2.6, “CSN Vote Window for the C++ API” for the C++ API, transactions processed between execution of AcceptTransaction and the final Receive that occurs after the SQL commit or rollback will have the same commit sequence number.

Figure 2.7, “CSN Vote Window for the C API” illustrates the vote window for the C API. Transactions processed between execution of the rtr_accept_tx call and the final rtr_receive_message call that occurs after the SQL commit or rollback will have the same commit sequence number.

Not all database managers require locking before the SQL commit operation. For example, some insert calls create a record only during the commit operation. For such calls, the application must ensure that the table or some token is locked so that other transactions are not incorrectly placed by RTR in the same commit group.

All database systems do locking at some level, at the database, file, page, record, field, or token level, depending on the database software. The application designer must determine the capabilities of whatever database software the application will interface with, and consider these in developing the application. For full use of RTR, the database your application works with must, at minimum, be capable of being locked at the record level.

Not all database managers require locking before the SQL commit operation. For example, some insert calls create a record only during the commit operation. For such calls, the application must ensure that the table or some token is locked so that other transactions are not incorrectly placed by RTR in the same commit group.

All database systems do locking at some level, at the database, file, page, record, field, or token level, depending on the database software. The application designer must determine the capabilities of whatever database software the application will interface with, and consider these in developing the application. For full use of RTR, the database your application works with must, at minimum, be capable of being locked at the record level.

2.4.4.1.2. Independent Transactions

When a transaction is specified as being independent (using the SetIndependentTransaction parameter set to true in the AcceptTransaction method (C++ API) or with the INDEPENDENT flag (C API)), the current commit sequence number is assigned to the independent transaction. Thus the transaction can be scheduled simultaneously with other transactions having the same CSN, but only after all transactions with lower CSNs have been processed.

RTR tracks time in cycles using windows; a vote window is the time between the close of one commit cycle and the start of the next commit cycle. For example, independent transactions include transactions such as zero-hour ledger posting (posting of interest on all accounts at midnight), and selling bets (assuming that the order in which bets are received has no bearing on their value).

RTR examines the vote sequence of transactions executing on the primary server, and determines dependencies between these transactions. The assumption is: if two or more transactions vote within a vote window, these transactions could be processed in any order and still produce the same result in the database. Such a group of transactions is considered independent of other transaction groups. Such groups of transactions that are mutually independent may still be dependent on an earlier group of independent transactions.

2.4.4.1.3. CSN Ordering

RTR tracks these groups through CSN ordering. A transaction belonging to a group with a higher CSN is considered to be dependent on all transactions in a group with a lower CSN. Because RTR infers CSNs based on run-time behavior of servers, there is scope for improvement if the application can provide hints regarding actual dependence. If the application knows that the order in which a transaction is committed within a range of other transactions is not significant, then using independent transactions is recommended. If an application does not use independent transactions, RTR determines the CSN grouping based on its observation of the timing of the vote.

2.4.4.1.4. CSN Boundary

To force RTR to provide a CSN boundary, the application must:

• Use dependent transactions. This is the default behavior.

• Ensure that a transaction is voted on after any transactions on which it is dependent.

The CSN boundary is between the end of one CSN and the start of the next, as represented by the last transaction in one commit group and the first transaction in the subsequent commit group.

In practice, for the transaction to be voted on after its dependent transactions, it is enough for the dependent transaction to access a common database resource, so that the database manager can serialize the transaction correctly.

Dependent transactions do not automatically have a higher CSN. To ensure a higher CSN, the transaction also needs to access a record that is locked by a previous transaction. This will ensure that the dependent transaction does not vote in the same vote cycle as the transaction on which it is dependent. Similarly, transactions that are independent do not automatically all have the same CSN. In particular, for the C API, if they are separated by an independent transaction, that transaction creates a CSN boundary.

RTR commit grouping enables independent transactions to be scheduled together on the shadow secondary. Flags on rtr_accept_tx and rtr_reply_to_client enable an application to signal RTR that it is safe to schedule this transaction for execution on the secondary within the current commit sequence group. In a shadow environment, an application can obtain certain performance improvements by using independent transactions where suitable. With independent transactions, transactions in a commit group can be executed on the shadow server in a different order than on the primary. This reduces waiting times on the shadow server.

For example, transactions in a commit group can execute in the order A2, A1, A3 or the primary partition and in the order A1, A2, A3 on the shadow site. Of course independent transactions can only be used where transaction execution need not be strictly the same on both primary and shadow servers. Examples of code fragments for independent transactions are shown in the code samples appendices of this manual.

Some of your applications may rely on batch processing for periodic activity. Application facilities can be created with batch processing. (The process for creating batch jobs is operating-system specific, and is thus outside the scope of this document.) Be careful in your design when using batch transactions. For example, accepting data in batch from legacy systems can have an impact on application results or performance. If such batch transactions update the same database as online transactions, major database inconsistencies or long transaction processing delays can occur.

The RTR journal on each node must be accessible to be used to replay transactions. When setting up your system, consider both journal sizing and how to deal with replay anomalies.

To size a journal, use the guidelines described in the section Creating a Recovery Journal in the VSI Reliable Transaction Router System Manager’s Manual.

Use of large transactions generally causes poor performance, not only for initial processing and recording in the database, but also during recovery. Large transactions fill up the RTR journals more quickly than small ones.

For replay anomalies, use the RTR_STS_REPLYDIFF status message to determine if a transaction has been recorded differently during replay. For details on this and other status messages, see the VSI Reliable Transaction Router C++ Foundation Classes manual or the VSI Reliable Transaction Router C Application Programmer’s Reference Manual.

You should also consider how the application is to handle secondary or shadow server errors and aborts, and write your application accordingly.

This information is roughly scalable to other CPUs and networks, and illustrates the concepts of RTR throughput on an untuned, as-shipped product and operating system. No guarantees are implied; however, these data may be used as a guide for baseline estimates.

These performance tests were run during April 2004 in the Nashua, N.H. RTR lab.

System Environment:

The tests were staged on an AlphaServer GS160 (four system building blocks (SBBs)), each with four EV6/7 730MHz CPUs. Only two of the SBBs were used in the tests and OpenVMS 7.3-1 was installed on both. Both were configured identically and were dedicated to these tests (there were no other users or processes).

The network interface card in each system was a 10/100Mbps twisted-pair Ethernet card (DEC602) enabled to run at 100Mbps. There was no other network traffic.

Test Scenario Overview:

The test goal was to provide information on RTR throughput as measured in transactions per second (TPS) and CPU load (%CPU) while varying four basic parameters: number of nodes (one and two), number of CPUs (one and quad), message size (100 to 60,000 bytes), and number of open channels (1 to 100).

This was accomplished with six test runs, varying appropriate parameters. The actual applications used to create and transfer transactions consisted of a client application, TPSREQ, and a server application, TPSSRV. These mini-applications are provided on the RTR software kit and use the RTR V4.2 C API.

RTR facilities were configured with the FE role running the client application, TPSREQ, on one node, while the router and backend roles were configured on the other. The backend was running the server application, TPSSRV.

The six test scenarios were:

Note that, in the following test runs, the charts indicate %CPU utilization and throughput in transactions per second (TPS). The %CPU utilization is the sum of RTRACP and application (TPSREQ, TPSSRV) CPU usage on all nodes in the facility.

To establish a baseline for network throughput, FTP (the File Transfer Protocol tool) was used to transfer a 9Mbyte file from one node to the other. This was done several times in both directions showing an average network transfer rate of about 9Mbytes/sec.

The results for a single node with an increasing number of channels are shown in Figure 2.8, “Single-Node, Single CPU, TPS and CPU Load by Number of Channels”

The results for a single node with four CPUs and an increasing number of channels are shown in Figure 2.9, “Single-Node, Quad CPU, TPS and CPU Load by Number of Channels”

The transactions used in these tests were regular read/write transactions; there was no use of optimizations such as READONLY or ACCEPT_FORGET.

The results of these tests using 100-byte messages suggest the following:

The results for a single node with a fixed number of channels (80) and increasing message size are shown in Figure 2.10, “Single-Node, Single CPU, TPS and CPU Load by Message Size” for a single CPU, and Figure 2.11, “Single-Node, Quad CPU, TPS and CPU Load by Message Size” for a quad CPU.

These tests using 80 client and server channels suggest that:

The results for the two-node single and quad CPU configurations are shown in Figure 2.12, “Two-Node, Single CPU TPS and CPU Load by Number of Channels” and Figure 2.13, “Two-Node, Quad CPU TPS and CPU Load by Number of Channels”.

This two-node test using 100-byte messages shows CPU usage with totals for frontend and backend combined (thus CPU utilization can be more than 100 percent). This test shows TPS increasing as the number of channels increases.

The two-node/quad-CPU test indicates that the constraint appears to be network bandwidth (values not shown in the figures) because the TPS rate flattens out at a network traffic level consistent with that measured on the same LAN by another independent test (using FTP to transfer data across the same network links). For example, using FTP on this Ethernet, multi-CPU configuration, shows transfer rates up to 8.9Mbytes per second, while using the RTR mini-applications shows transfer rates of about 6.4Mbytes per second. Thus improving network bandwidth could likely improve TPS rates when using RTR.

Use concurrent servers in database applications to optimize performance and continue processing when a concurrent server fails.

Partitioning data enables the application to balance traffic to different parts of the database on different disk drives. This achieves parallelism and provides better throughput than using a single partition. Using partitions may also enable your application to survive single-drive failure in a multi-drive environment more gracefully. Transactions for the failed drive are logged by RTR while other drives continue to record data.

To achieve performance goals, you should establish facilities spread across the nodes in your physical configuration using the most powerful nodes for your backends that will have the most traffic.

In some applications with several different types of transactions, you may need to ensure that certain transactions go only to certain nodes. For example, a common type of transaction is for a client application to receive a stock sale transaction, which then proceeds through the router to the current server application. The server may then respond with a broadcast transaction to only certain client applications. This exchange of messages between frontends and backends and back again can be dictated by your facility definition of frontends, routers, and backends.

Placement of routers can have a significant effect on your system performance. With connectivity over a wide-area network possible, do not place your routers far from your backends, if possible, and make the links between your routers and backends as high speed as possible. However, recognize that site failover may send transactions across slower-speed links. For example, Figure 2.14, “Two-Site Configuration” shows high-speed links to local backends, but lower-speed links that will come into use for failover.

Additionally, placing routers on separate nodes from backends provides better failover capabilities than placing them on the same node as the backend.

In some configurations, you may decide to use a dual-rail or multihome setup for a firewall or to improve network-related performance. (See the VSI Reliable Transaction Router System Manager’s Manual section on Network Transports for information on this setup.)

When a server or client application sends out a broadcast message, the message passes through the router and is sent to the client or server application as appropriate. A client application sending a broadcast message to a small number of server applications will probably have little impact on performance, but a server application sending a broadcast message to many, potentially hundreds of clients, can have a significant impact. Therefore, consider the impact of frequent use of large messages broadcast to many destinations. If your application requires use of frequent broadcasts, place them in messages as small as possible. Broadcasts could be used, for example, to inform all clients of a change in the database that affects all clients.

Figure 2.15, “Message Fan-Out” illustrates message fan-out from client to server, and from server to client.

You can also improve performance by creating separate facilities for sending broadcasts.

Very large configurations with unstable or slow network links can reduce performance significantly. In addition to ensuring that your network links are the fastest you can afford and put in place, examine the volume of inter-node traffic created by other uses and applications. RTR need not be isolated from other network and application traffic, but can be slowed down by them.

Read-only transactions can significantly improve throughput because they do not need to be journaled. A read-only database can sometimes be updated only periodically, for example, once a week rather than continuously, which again can reduce application and network traffic.

When using transactional shadowing, it can enhance performance to process certain transactions as independent. When transactions are declared as independent, processing on the shadow server proceeds without enforced serialization. Your application analysis must establish what transactions can be considered independent, and you must then write your application accordingly. For example, bets placed at a racetrack for a specific race are typically independent of each other. In another example, transactions within one customer's bank account are typically independent of transactions within another customer's account. For examples of code snippets for each RTR API, see the appendices of samples in this manual.

For security purposes, your application transactions may need to pass through firewalls in the path from the client to the server application. RTR provides this capability within the CREATE FACILITY syntax. See the VSI Reliable Transaction Router System Manager’s Manual, Network Transports, for specifics on how to specify a node to be used as a firewall, and how to set up your application tunnel through the firewall.

Nodes in your configuration are often specified with names and IP or DECnet addresses fielded by a name server. When the name server goes down or becomes unavailable, the name service is not available and certain requests may fail. To minimize such outages, declare the referenced node name entries in a local host names file that is available even when the name server is not. Using a host names file can also improve performance for name lookups. For details on this, see the VSI Reliable Transaction Router System Manager’s Manual section on Network Transports.

Operations staff often create batch or command procedures to take snapshots of system status to assist in monitoring applications. The character cell displays (ASCII output) of RTR can provide input to such procedures. Be aware that system responses from RTR can change with each release, which can cause such command procedures to fail. If possible, plan for such changes when bringing up new versions of the product.

RTR expects server applications to be transaction aware; an application must be able to roll back an appropriate amount of work when asked. Furthermore, to preserve transaction integrity, rollback must be all or nothing. Each transaction incurs some overhead, and the application must be prepared to deal with failures and concomitant rollback gracefully. When designing your client and server applications, note the outcome of transactions. Transactional applications often store data in variables that pertain to the operation taking place outside the control of RTR. Depending on the outcome of the RTR transaction, the values of these variables may need to be adjusted. RTR guarantees delivery of messages (usually to a database), but RTR does not know about any data not passed through RTR.

The rule is:

Code your application to preserve transaction integrity through failures.

The client and server applications must not exchange any data that makes sense on only one node in the configuration. Such data can include, for example, a memory reference pointer, whose purpose is to allow the client to reference this context in a later transaction, indexes into files, node names, or database record numbers. These values only make sense on the machine on which they were generated. If your application sends data to another machine, that machine will not be able to interpret the data correctly. Furthermore, data cannot be shared across servers, transaction controllers, or channels.

The rule is: How you track state must be meaningful on all nodes where your application runs.

Transactions are assumed to contain all the context information required to be successfully executed. An RTR transaction is assumed to be independent of time of processing. For example, in a shadow environment, if the secondary server cannot credit an account because it is past midnight, but the transaction has already been successfully committed on the primary server, this would cause an inconsistency between the primary and secondary databases. Or, in another example, Transaction B cannot rely on the fact that Transaction A performed some operation before it.

Make no assumptions about the amount of time that will occur between transactions, and avoid using a transaction to establish a session with a server application that can time out. Such a timeout might occur in a client application that logs onto a server application that sets a timer to determine when to log the client off. If a crash occurs after a successful logon, subsequent transactions may fail because the logon session is no longer valid.

The rule is:

If you have operations that must not be shadowed, identify them and exclude them from your application. Furthermore, do not keep a state that can become stale over time.

In your application, you can define transactions as independent with the C++ API, using the SetIndependentTransaction method in your transaction controller AcceptTransaction or SendApplicationMessage calls. Using the C API, you use the independent transaction flag in your rtr_accept_tx or rtr_reply_to_client calls.

For more information on the independent transaction methods in the RTRServerTransactionController class, refer to the VSI Reliable Transaction Router C++ Foundation Classes manual. For more information on the independent transaction flag and the different uses of these calls, refer to the VSI Reliable Transaction Router C Application Programmer’s Reference Manual.

Shadow server use is aimed at keeping two identical copies of the database synchronized. For example, Figure 3.1, “Transactional Shadow Servers” illustrates a configuration with a router serving two backends to two shadow databases. The second router is for router failover.

If an update of a copy triggers the update of a third common database, the application must determine whether it is running as a primary or a secondary, and only perform an update if it is the primary. Otherwise, there can be complex failure scenarios where duplication can occur.

For example, RTR has no way to determine if a transaction being shadowed is a one-time-only transaction, such as a bookstore debiting your credit card for the purchase of a book. If this transaction is processed on the primary node and the processed data fed to a third common database, and the transaction is later processed on the secondary node, your account would incorrectly be double charged. The application must handle this situation correctly.

The rule is:

Design your application to deal correctly with transactions, such as debiting a credit card or bank account, that must never be performed more than once.

Figure 3.2, “Shadow Servers and Third Common Database (not recommended)” shows a configuration with two shadow servers and a third, independent server for a third, common database. This is not a configuration recommended for use with RTR without application software that deals with the kind of failure situation described above. Another method is to decouple the shadow message from the other branch.

When updating a single resource through multiple paths, the recommended method is to use the RTR standby functionality.

All information required to process a transaction from the perspective of the server application should be contained within the transaction message. For example, if the application required a user-ID established earlier to successfully execute the transaction, the user-ID should be included in the transaction message.

The rule is:

Construct complete transaction messages within your application.

While a server application is processing a transaction, and particularly before it "accepts" the transaction, it must ensure that all shared resources accessed by that transaction are locked. Failure to do so can cause unpredictable results in shadowing or recovery.

The rule is:

Lock shared resources while processing each transaction.

For the atomic attribute, the result of a transaction is all or nothing, that is, either totally committed or totally rolled back. To ensure atomicity, do not use a data manager that cannot roll back its updates on request. All standard data managers or database management systems have the atomicity attribute. However, in some cases, when interfacing to an external legacy system, a flat-file system, or an in-memory database, a transaction may not be atomic.

For example, a client application may believe that a transaction has been rejected, but the database server does not. With a database manager that can make this mistake, the application itself must be able to generate a compensating transaction to roll back the update.

Data managers that do not use XA/DTC, DECdtm or Microsoft DTC to integrate with RTR using XA or DECdtm must be programmed to handle rtr_mt_msg1_uncertain messages.

For example, to illustrate the atomicity rules, Figure 3.3, “Uncertain Interval for Transactions” shows the uncertain interval in a transaction sequence that the application program must be aware of and take into account, by performing appropriate rollback.

If there is a crash before the AcceptTransaction method (rtr_accept_tx statement for the C API) is executed, on recovery, the transaction is replayed as rtr_mt_msg1 because the database will have rolled back the prior transaction instance. However, if there is a crash after the AcceptTransaction method or rtr_accept_tx statement is executed, on recovery, the transaction is replayed as rtr_mt_msg1_uncertain because RTR does not know the status of the prior transaction instance. Your application must understand the implications of such failures and deal with them appropriately.

A transaction either creates a new and valid state of data, or, if any failure occurs, returns all data to its state as it was before the start of the transaction. This is called consistency.

RTR commit grouping allows independent transactions to be scheduled together on the shadow secondary.

For a transaction to be durable, the changes caused by transaction commitment must survive subsequent system and media failure. Thus transactions are both persistent and stable.

For example, your bank deposit is durable if, once the transaction is complete, your account balance reflects what you have deposited.

If there are dependencies between separate RTR transactions, these should be considered carefully because the locking mechanisms of resource managers can cause unexpected behavior. These issues around locking mechanisms occur only if there is more than one server for the same partition.

For example, consider the case where there is a transaction T1 which inserts a record in the database and a subsequent transaction T2 which uses that record to make another update. If the partition has been configured with concurrent servers, it can happen that the update transaction T2 which has been given to a free server will begin executing and reach the database before the insert operation issued by transaction T1 has completed the commit process. In this scenario, the inserted record is not yet visible to the update transaction T2 because the commit is not yet complete. This will cause transaction T2 to fail. However, if the database table being updated is locked for the duration of the insert, transaction T2 will block (wait) until the insert has committed and there will be no possibility of transaction T2 overtaking transaction T1.

In another example, the first transaction T1 makes an update to the table and a second transaction T2 uses the updated value in its transaction. If the resource manager does not lock the row being accessed by transaction T1 right at the start of the update, that row can be queried by the second transaction T2 which has started on a concurrent server. However, transaction T2 will in this case be working with the old and not the updated value that was the result of T1. To prevent such unexpected and potentially undesirable behavior, check the locking mechanisms of the resource managers being used before using concurrent servers.

RTR provides client applications the option to specify a transaction timeout, but has no provision for server applications to specify a timeout on transaction duration. If there is a scarcity of server application processes, all other client transactions remain queued. If these transactions have also specified timeouts, they are aborted by RTR (assuming that the timeout value is less than 2 minutes).

The first (and easier) option is to use concurrent server processes. This allows transaction requests to be serviced by other free servers, even if one server is occupied by such a transaction that is taking a long time to disappear. The second option is to design the server application so that it can abort the transaction independently.

There are three cases where this use of concurrent servers is not ideal. First, there is an implicit assumption about how many such lingering transactions might remain on the system. In the worst case, this could exceed or equal the number of client processes. But having so many concurrent server processes to cater to this contingency is wasteful of system resources. Second, use of concurrent servers is beneficial when the servers do not need to access a common resource. For instance, if all these servers needed to update the same record in the database, they would simply be waiting on a lock taken by the first server. Additional servers do not resolve this issue. Third, it must make business sense to have additional servers. For example, if transactions must be executed in the exact order in which they entered the system, concurrent servers may introduce sequencing problems.

Take the example of the order matcher in a stock trading application. Business rules may dictate that orders be matched on a first-come, first- matched basis; using concurrent servers would negate this rule.

The second option is to let the server application process administer its own timeout and abort the transaction when it sees no activity on its input stream.

To ensure that transactions are fully executed and that the database is consistent, RTR uses the two-phase commit process for committing a transaction. The two-phase commit process has both a prepare phase and a commit phase. Transactions must reach the commit phase before they are hardened in the database.

The two-phase commit mechanism is initiated by the client when it executes a call to RTR that declares that the client has accepted the transaction. The servers participating in the transaction are then asked to be prepared to accept or roll back the transaction, based on a subsequent request.

With RTR, client and server applications communicate by exchanging messages in a transaction dialog. Transactional messages are grouped in a unit of work called a transaction. RTR takes ownership of a message when called by the application.

A transaction is a group of logically connected messages exchanged in a transaction dialog. Each dialog forms a transaction in which all participants have the opportunity to accept or reject the transaction. A transaction either commits or aborts. When the transaction is complete, all participants are informed of the transaction's completion status. The transaction succeeds if all participants accept it, but fails if even one participant rejects it.

In the context of a transaction, an RTR client application sends one or more messages to the server application, which responds with zero or more replies to the client application. Client messages can be grouped to form a transaction. All work within a transaction is either fully completed or all work is undone. This ensures transaction integrity from client initiation to database commit with the cooperation of the server application.

For example, say you want to take $20 from your checking account and add it to your savings account. With an application using RTR you are assured that this entire transaction is completed; you will not be left at the point where you have taken $20 from your checking account but it has not yet been deposited in your savings account. This feature of RTR is transactional integrity, illustrated in Figure 3.5, “Transactional Messaging”.

The transactional message is either all or nothing for everything enclosed in brackets [ ] in Figure 3.5, “Transactional Messaging”.

An RTR client application sends one or more messages to one or more server applications and receives zero or more responses from one or more server applications. For example:

RTR generates a unique identifier, the transaction ID or TID, for each transaction. The client can inject also its own TID into RTR. Doing so will make RTR treat the transaction as a nested transaction.

Figure 3.6, “Transactional Messaging Interaction” illustrates frontend/backend interaction with pseudo-code for transactions and shows transaction brackets. The transaction brackets show the steps in completing all parts of a transaction, working from left to right and top to bottom. The transaction (txn) is initiated at "Start txn" at the frontend, and completed after the "Commit txn" step on the backend. The transaction ID is encoded to ensure its uniqueness throughout the entire distributed system. In the prepare phase on the server, the application should lock the relevant database (db) records. The commit of a transaction hardens the commit to the database. The rtr_start_tx message specifies the characteristics of the transaction. RTR identifies the server based on key information in the transaction.

Broadcast messaging lets client and server applications send non- transactional information to recipients. Recipients must be declared with the opposite role; that is, a client can send a broadcast message to servers, and a server can send a broadcast message to clients. Broadcasts are delivered on a "best try" basis; they are not guaranteed to reach every potential recipient. A broadcast message can be directed to a specific client or server, or be sent to all reachable recipients.

This point-to-point messaging using broadcast semantics is a feature to use instead of transactions when the information being sent is not recorded as a transaction in the database, and when you need to send information to several clients (or servers) simultaneously. For example, in a stock trading environment, when a trade has been completed and the stock price has changed, the application can use a broadcast message to send the new stock price to all trading stations. Another use for such messages is to inform the applications about a state change in the environment (for example, the fact that the exchange is now closed for business).

Event numbers are provided as a list beginning with RTR_EVTNUM_USERDEF and ending with RTR_EVTNUM_ENDLIST. To subscribe to all user events, an application can use the range indicators RTR_EVTNUM_USERBASE and RTR_EVTNUM_USERMAX, separated by RTR_EVTNUM_UP_TO, to specify all possible user event numbers.

A user broadcast is named or unnamed. An unnamed broadcast does a match on user event number; the event number completely describes the event. A named broadcast does a match on both user event number and recipient name. The recipient name is a user-defined string. Named broadcasts provide greater control over who receives a particular broadcast.

Named events specify an event number and a textual recipient name. The name can include wildcards (% and *).

For all unnamed events specify the evtnum field and RTR_NO_RCPSPC as the recipient name.

RTR delivers status information to which client and server applications can subscribe. Status information is delivered as messages, where the type of each message is an RTR event.

RTR events are numbered. The base value for RTR events is defined by the symbol RTR_EVTNUM_RTRBASE; its maximum value is defined by the symbol RTR_EVTNUM_RTRMAX. RTR events and event numbers are listed in the Reliable Transaction Router API manuals and in the RTR header files rtr.h and rtrapi.h.

An application can subscribe to RTR events to receive notification of external events that are of interest to the application. For example, a shadow server may need to know if it is a primary or a secondary server to perform certain work, such as uploading information to a central database, that is done at only one site.

To subscribe to all RTR events, use the range indicators RTR_EVTNUM_RTRBASE and RTR_EVTNUM_RTRMAX. RTR events are delivered as messages of type rtr_mt_rtr_event.

In application design, consider creating separate facilities for sending broadcasts. By separating broadcast notification from transactional traffic, performance improvements can be substantial. Facilities can further be reconfigured to place the RTR routers strategically to minimize wide-area traffic.

A server application can expect to see a primary or secondary event delivered only in certain transaction states. For more detail, see the state diagrams in Appendix C, Server States.

Neither txn1 nor txn2 can proceed; they are in a deadly embrace. Figure 3.7, “Deadly Embrace” illustrates a deadly embrace.

RTR attempts to resolve deadlocks by aborting one deadlocked transaction path with error code RTR_STS_DLKTXRES and replaying the transaction. Other paths are not affected. Server applications should be written to handle this status appropriately.

The RTR status code RTR_STS_DLKTXRES can occur under several environmental conditions that RTR detects and works around. The application need not take any explicit action other than releasing the resources connected with the active transaction such as doing a rollback on the database transaction.

As an example of the first case, consider clients A and B both performing transactions TCA and TCB, where both TCA and TCB include a message to both server X and server Y followed by an ACCEPT. There is only one instance of Server X and Server Y available, and due to the quirks of distributed processing, only Server X receives the message belonging to TCA and only Server Y receives the message belonging to TCB. Figure 3.8, “Scenario for Distributed Deadlock” reflects this scenario. Because Server Y has no chance of accepting TCA until TCB is processed to completion and Server X has no chance of accepting TCB until TCA is processed to completion, Server X and Y are in a distributed deadlock. In such a case, RTR selects TCA or TCB to abort with DLKTXRES and replays it in a different order.

Sometimes RTR needs to abort a transaction and reschedule it. For example, it can happen that a state change is needed after the primary server started to process a transaction but RTR had to change its role to secondary before the transaction was completed. Thus the transaction would be executed on the other node as primary and later played to this server as secondary. RTR uses the same status code RTR_STS_DLKTXRES when aborting the transaction.

One method for improving response time is to send multiple messages from clients without waiting for a reply. The messages can be sent to different partitions to provide parallel processing of transactions.

For certain read-only applications, RTR can be used without shadowing to establish sites to protect against site failure. The method is to define multiple non-overlapping facilities with the same facility name across a network that is geographically dispersed. In the facility, define a failover list of routers, for example, some in one city, some in another. Then when the local router fails, a client is automatically reconnected to another node. If all local nodes in the facility are unavailable, the client is automatically connected to a node at the alternate site.

Another method is to define a partition on a standby server for read-only transactions. This minimizes network traffic to the standby. A read-only partition on a standby server can reduce node-to-node transaction locking.

Data-content routing is the capability provided by RTR to support the partitioned data model. With the C++ API, you define partitions with the RTRPartitionManager class. Partitions are defined with partition name, facility name and KeySegment attributes. Using RTR data-content routing, message content determines message destination. The routing key, defined in the C++ API RTRKeySegment class is embedded within the application message. When a server is started, the server informs RTR that it serves a particular partition.

When a client sends a message that references that particular partition, RTR knows where to find a server for that partition and routes the message accordingly. Even if the server is moved to another location, RTR can still find it. The client and the application do not need to worry about locating the server. This is location independence.

The benefits of data-content routing are simpler application development and flexible, scalable growth without application changes. RTR hides the underlying network complexity from the application programmer, thus simplifying development and operation.

With Set State methods within the RTRServerTransactionProperties class, users can change a transaction state in their application servers. With the event-driven processing model, your states are shown through the message handlers. In the polling model, the states are accessed with an RTRData::GetMessageType() call.

Consider the following scenario: upon receiving an rtr_mt_accepted message indicating that a transaction branch is committed, the server normally performs an SQL commit to write all changes to the underlying database. However, the server that is to detect the SQL commit or the underlying database may be temporarily unavailable.

With the C++ API, you can change the state of a transaction to EXCEPTION using the SetStateToException method, to temporarily put this transaction in the exception queue. When things get better, users can call SetStateToCommit or SetStateToDone to change the transaction state back to COMMIT or DONE, respectively.

The following example shows how a transaction state can be changed using the set state methods. See the RTRServerTransactionProperties class documentation in the VSI Reliable Transaction Router C++ Foundation Classes manual for more details.

(RTRServerTransactionProperties::SetStateToException();)
RTRServerTransactionProperties_object_name.SetStateToException(stCurrentTxnState);)

The parameter stCurrentTxnState is a transaction state of type rtr_tx_jnl_state_t.

Normally at the RTR command level, users need to provide at least facility name and partition name qualifiers to help RTR select a desired set of transactions to be processed. Because a transaction's TID (rtr_tid_t) is unambiguously unique, a user needs only to specify the transaction's current state and its TID.

Note that if a transaction has multiple branches running on different partitions simultaneously on this node, RTR will reject this set transaction request with an error. RTR can only change state for one branch of a multiple partition transaction at a time.

Applications should include code for all RTR expected return message types. Message types are returned to the application in the message status block. For more detail on message types, see the VSI Reliable Transaction Router C Application Programmer’s Reference Manual.

Message processing involves a message being sent by a client and a reply coming back from one or more servers. This is where the business application processing takes place.

Figure 4.4, “Accept Processing” illustrates the accept-processing portion of a completed RTR transaction.

The steps in accept processing are:

If transaction timeouts are not required, the transaction starts on the next SendMessage call.

When a message is received, the message status block contains the transaction identifier (TID).

You can use the GetTID call to obtain the RTR transaction identifier for the current transaction. This identifier is a unique number generated by RTR for each transaction. The application can use the TID if the client needs to know the TID to take some action before receiving a response.

The AcceptTransaction call by the client begins the prepare phase of the two- phase commit protocol. An accepted transaction is not complete until a message of type rtr_mt_accepted or rtr_mt_rejected is received.

The application can specify a reason on the AcceptTransaction method so that the caller can specify an accept reason that is passed on to all participants in the transaction. If more than one transaction participant specifies a reason, the reason values are ORed together by RTR. The accept is final: the caller cannot reject the transaction later. The caller cannot send any more messages for this transaction.

A client can accept a transaction in one of two ways: with the AcceptTransaction call or by using the SetAcceptTransaction method. Using the SetAcceptTransaction method removes the need to issue an AcceptTransaction method and can help optimization of client traffic. Merging the data and accept messages in one call puts them in a single network packet. This can make better use of network resources and improve throughput.

Any participant in the transaction can call RejectTransaction. The reject is final and it is impossible for the caller to accept the transaction later. The RejectTransaction method rejects a transaction. Once the transaction has been rejected, the caller receives no more messages for this transaction.

The server can call the ForceRetry method to have RTR redeliver the transaction beginning with rtr_mt_msg1 without aborting the transaction for other participants. Using the RejectTransaction method, the application can specify a reason that the caller can pass on to all participants in the transaction. If more than one transaction participant specifies a reason, the reason values are ORed together by RTR.

A server application can end a transaction by either accepting or rejecting the transaction. RTR can reject a transaction at any time after the transaction is started but before it is committed. For example, if RTR cannot deliver a transaction to its destination, it rejects the transaction and delivers the reject completion status to all participants that know about the transaction.

A transaction is accepted explicitly with the AcceptTransaction method, and rejected explicitly with the RejectTransaction method. RTR can reject a transaction at any time once the transaction is started but before it is committed. If RTR cannot deliver a transaction to its destination, it rejects the transaction explicitly and delivers the reject completion status to all participants.

A transaction participant can specify a reason for an accept or reject on the AcceptTransaction and RejectTransaction methods. If more than one transaction participant specifies a reason, RTR uses the OR operator to combine the reason values together. For example, with two servers, A and B, each providing a reason code of 1 or 2, respectively, the client receives the result of the OR operation, reason code 3, in its message buffer in RTRData.

A transaction is done once a client or server application receives a completion message, either an rtr_mt_accepted or rtr_mt_rejected message from RTR. An application no longer receives messages related to a transaction after receiving a completion message or if the application uses RejectTransaction A client or server can also specify SetForgetTransaction to signal its acceptance and end its involvement in a transaction early. RTR returns no more messages (including completion messages) associated with the transaction; any such messages received will be returned to the caller.

A client or server application no longer receives messages related to a transaction after it receives an OnAccepted or OnRejected message from RTR, or if the application called RejectTransaction.

For more information on RTRTransactionController methods, refer to the VSI Reliable Transaction Router C++ Foundation Classes manual.

RTR provides a relatively simple way to administer a transaction timeout in the server. Use of timeout values on the Receive() method lets a server application specify how long it is prepared to wait for the next message. (Of course, the server should be prepared to wait forever to get a new transaction or for the result of an already-voted transaction.)

One way to achieve this would be to have a transaction controller-specific global variable, say, called SERVER_INACTIVITY_TIMEOUT, which is set to the desired value (in milliseconds-that is, use a value of 5000 to set a 5-second timeout). Note that this timeout value should be used after receiving the first message of the transaction. The value should be reset to RTR_NO_TIMOUTMS after receiving the rtr_mt_prepare message. Whenever the Receive method completes with an RTR_STS_TIMOUT, the server transaction controller calls RejectTransaction to abort the partially processed transaction. This would prevent transactions from occupying the server process beyond a reasonable time.

A prepared application votes its intention using the AcceptTransaction method. An application that does not agree to commit to this transaction votes with the RejectTransaction method. This is the first (or prepare) phase of the two-phase commit process.

The impact of a crash on application recovery depends on where in the process the crash occurs. RTR handles the recovery, with the assistance of the application.

RTR keeps track of how many times a transaction is presented to a server application before it is VOTED. The rule is: three strikes and you're out! After the third strike, RTR rejects the transaction with the reason RTR_STS_SRVDIED. The server application has committed the transaction and the client believes that the transaction is committed. The transaction is flagged as an exception and the database is not committed. Such an exception transaction can be manually committed if necessary. This process eliminates the possibility that a single rogue transaction can crash all available copies of a server application at both primary and secondary sites.

Application design can change this behavior. The application can specify the retry count to use when in recovery using the SetRecoveryRetryCount method in the RTRBackEndPartitionProperties class, or the system administrator can set the retry count from the RTR CLI with the SET PARTITION command. If no recovery retry count is specified, RTR retries replay three times. For recovery, retries are infinite. For more information on the SET PARTITION command, refer to the VSI Reliable Transaction Router System Manager’s Manual; for more information on the SetRecoveryRetryCount method, refer to the VSI Reliable Transaction Router C++ Foundation Classes manual.

When a node is down, the operator can select a different backend to use for the server restart. To complete any incomplete transactions, RTR searches the journals of all backend nodes of a facility for any transactions for the key range specified in the server's RegisterPartition call.

RTR provides two mechanisms for coordinating RTR transactions with database transactions (or database managers): transaction management coordination (XA, DECdtm) and RTR replay technology.

To use broadcast messaging with the C++ API, client and server applications can overload their event handlers.

To enable communication between two applications of the same type, create a second transaction controller of the other type. Messaging destination names can include wildcards, enabling flexible definition of the subset of recipients for a particular broadcast.

Use the SendApplicationEvent method to broadcast a user-event message. Broadcast types include user events and RTR events; both are numbered and can be named.

SendApplicationEvent( evUserEventNumber, "*", RTR_NO_MSGFMT)

Receive(
...pmsg,maxlen,
   pmsgsb);

rtr_status_t Receive( *pRTRData )

RTR callout servers enable security checks to be carried out on all requests using a facility. Callout servers can run on backend or router nodes. They receive a copy of every transaction delivered to or passing through the node, and they vote on every transaction. To enable a callout server, use the /CALLOUT qualifier when issuing the RTR CREATE FACILITY command. Callout servers are facility based, not key-range or partition based.

  RTRFacilityManager.CreateFacility(...
                                   bEnableRouterCallout = true
                                     ...);

  RTRFacilityManager.CreateFacility(...
                                   bEnableBackendCallout = true);

The rtr.h file included with the product defines all RTR data, status, and message types, including text that can be returned in error messages from an application. You must use it in application compilation.

To support the multi-operating system environment, error codes are processed by RTR using data values in rtr.h, and translated into text messages. Status codes are described in the VSI Reliable Transaction Router C Application Programmer’s Reference Manual.

The RTR journal is always in use, recording transaction activity to persistent storage. The journal thus provides the capability of recovery from any single hardware or process failure. When a server application no longer provides service, for example, when it goes off line, goes down, or is taken out of service temporarily, RTR aborts all transactions for that service with RTR_STS_NODSTFND. When doing transactional shadowing, the journal at the active site is used to record transactions, for the out-of- service site. Journaling on frontends is required to support nested transactions to record the final state of a transaction.

If transactions do not update the database, specify them as read-only by using the RTR_F_SEN_READONLY flag on the rtr_send_to_server call. Read-only transactions are not recovered as uncertain transactions. Also, in a shadowed environment, these transactions would not be remembered and would therefore save on journal space.

RTR enables an application to route transactions by partition or key range, rather than sending all transaction requests to a single server application.

When RTR has chosen a specific server for a request within a given transaction, RTR ensures that all requests within that transaction are routed to the same server application channel.

Key ranges, or data partitions, are a major concept in RTR. An application is said to service a partition. Failovers and other availability attributes are applied to all applications, which service a given partition. All server applications able to service a specific data partition on a given node are called concurrents of one another. Concurrent servers may be either multiple channels within a given process, or separate processes.

Partitions can be given names. This lets the system manager manipulate the attributes for a partition at runtime. For example, the operator can temporarily suspend the presentation of online transactions to the partition. This provides time to initiate a database backup operation.

It is possible to give a different name to the same partition on each backend. However, this is not recommended, because RTR Explorer displays the partition name when grouping backends by partition. If a partition is not given the same name on all backends, RTR Explorer will display a string representation of the key range instead of one of the names.

Partitions, in RTR, are the essence of routing. The server application declares its intention to service a certain partition or key range by associating itself with a name, or explicitly defining the key format and range of values.

To plan for future expansion, consider using compound keys rather than single field keys. For example, using a bank example, with a bank that has multiple branches, an application that routes data to each bank can use a BankID key field or partition. Over time, the application may also need to further subdivide transactions not only by bank but also by customer ID. If the application is initially written with a compound key providing both a BankID and a CustomerID key, it can be simpler to make such a change in future.

  ... status = rtr_receive_message(&Channel,
                                           RTR_NO_FLAGS,
                                           RTR_ANYCHAN,
                                           MsgBuffer,
                                           DataLen,
                                           RTR_NO_TIMOUTMS,
                                           &MsgStatusBlock);

5.4.1.3. RTR Message Types

RTR calls and responses to them contain RTR message types (mt) such as rtr_mt_reply or rtr_mt_rejected. The four groups of message types, listed in Table 5.4, “RTR Message Types”, are:

• Transactional

• Transactional status

• Event-related

• Informational

Transactional	Status	Event-related	Informational
rtr_mt_msg1	rtr_mt_accepted	rtr_mt_user_event	rtr_mt_opened
rtr_mt_msg1_uncertain	rtr_mt_rejected	rtr_mt_rtr_event	rtr_mt_closed
rtr_mt_msgn	rtr_mt_prepare		rtr_mt_request_info
rtr_mt_reply	rtr_mt_prepared		rtr_mt_rettosend

Applications should include code for all RTR expected return message types. Message types are returned to the application in the message status block. For more detail on message types see the VSI Reliable Transaction Router C Application Programmer’s Reference Manual. Figure 5.1, “RTR C API Calls for Transactional Messaging” shows the RTR C API calls you use to achieve transactional messaging in your application.

 status = rtr_send_to_server(p_channel,
                             RTR_NOFLAGS,
                             &txn_msg,
                             msg_size,
                             TXN_MSG_FMT );

The XA software architecture of RTR provides interoperability with the Distributed Transaction Controller of Microsoft, MS DTC. Thus RTR users can develop application programs that update MS SQL databases, MS MQ, or other Microsoft resource managers under the control of a true distributed transaction. RTR as a distributed transaction manager communicates directly with MS DTC to manage a transaction or perform recovery using the XA protocol. For each standard XA call received from RTR, MS DTC translates it into a corresponding OLE transaction call that SQL Server or MS MQ expects to perform database updates. This is shown in Figure 5.2, “MS DTC and RTR”.

Both the resource manager instance name and the database (RM) name in [OPEN- STRING] must be identical to that in the previously executed REGISTER RM command. The information is stored in the RTR key segment structure, and the RTR_F_OPE_XA_MANAGED flag associates the channel with the XA interface.

Only one transaction at a time is processed on an RTR channel; thus a server process or thread of control can only open one channel to handle a single XA request. Better throughput can be achieved by using a multithreaded application.

For example, the following code from a sample server application shows use of the RM key, the XA flag, and commenting out commits and rollbacks for the Oracle and DTC environments.

This parameter points to a valid channel identifier when the application receives an rtr_mt_opened message.

The facility name is a required string supplied by the application. It identifies the RTR facility used by the application. The default facility name for the RTR CLI only is RTR$DEFAULT_FACILITY; there is no default facility name for an RTR application. You must supply one.

rtr_facnam_t
facnam = "DESIGN";

rtr_evtnum_t all user_events[]={
    RTR_EVTNUM_USERDEF,
        RTR_EVTNUM_USERBASE,
        RTR_EVTNUM_UP_TO,
        RTR_EVTNUM_USERMAX,
    RTR_EVTNUM_ENDLIST
};

There are both RTR events and user events. For additional information on employing events, see the Broadcast Messaging Processes section of this chapter, and the section on RTR Events in the VSI Reliable Transaction Router C Application Programmer’s Reference Manual.

You can use the facility access key to restrict client or server access to a facility. The key acts as a password to restrict access to the specific facility for which it is declared.

  rtr_access_t
   access = "amalasuntha";

The facility access key is a string supplied by the application. The first server channel in an RTR facility defines the access key; all subsequent server and client open channel requests must specify the same access value. To use no access key, use RTR_NO_ACCESS or NULL for the access argument.

You can also use this feature for version control. By changing the access code whenever an incompatible protocol change is made in the application message format, client applications are prevented from processing transactions with the server applications.

To specify the key range for a partition to do data-content routing, the server application defines the routing key when it opens a channel on a facility with the rtr_open_channel call. All servers in a facility must specify the same offset, length, and data type for the key segments in the rtr_open_channel call; only high and low bounds ( *ks_lo_bound, *ks_hi_bound) can be unique to each server key segment. By application convention, the client places key data in the message at the offset, length, and data type defined by the server.

The channel-open operation completes asynchronously. Call completion status is returned in a subsequent message. RTR sends a message to the application indicating successful or unsuccessful completion; the application receives the status message using an rtr_receive_message call. If status is rtr_mt_opened, the open operation is successful. If status is rtr_mt_closed, the open operation is unsuccessful, and the application must examine the failure and respond accordingly. The channel is closed.

case rtr_mt_opened:
  printf(" Channel %d opened\n", channel);
  status = RTR_STS_OK;
  break;
case rtr_mt_closed:
  p_status_data = (rtr_status_data_t *)txn_msg;
  printf(" cannot open channel because %s\n",
   rtr_error_text(p_status_data->status));
  exit(-1);

Use the call rtr_error_text to find the meaning of returned status. A client channel will receive no message at all if a facility is configured but no server is available. Once a server becomes available, RTR sends the rtr_mt_opened message.

To close a channel, the application uses the rtr_close_channel call. A channel can be closed at any time after it has been opened. Once closed, no further operations can be performed on a channel, and no further messages for the channel are received.

To receive on a channel and obtain status information from RTR, use the rtr_receive_message call. To receive on any open channel, use the RTR_ANYCHAN value for the p_rcvchan parameter in the rtr_receive_message call. To receive from a list of channels, use the p_rcvchan parameter as a pointer to a list of channels, ending the list with RTR_CHAN_ENDLIST. An application can receive on one or more opened channels. RTR returns the channel identifier. A pointer to a channel is supplied on the rtr_open_channel call, and RTR returns the channel identification (ID) by filling in the contents of that pointer.

{
rtr_receive_message (&channel, RTR_NO_FLAGS, RTR_ANYCHAN, txn_msg, maxlen,
RTR_NO_TIMOUTMS,
MsgStatusBlock);
} . . .
   typedef struct {
   rtr_msg_type_t  msgtype;
   rtr_usrhdl_t      usrhdl;
   rtr_msglen_t     msglen;
   rtr_tid_t  tid;
   rtr_evtnum_t     evtnum;
   } rtr_msgsb_t;

RTR delivers both RTR and application messages when the rtr_receive_message call completes. The application can use the message type indicator in the message status block to determine relevant buffer format. For further details on using message types and interpreting the contents of the user buffer, refer to the VSI Reliable Transaction Router C Application Programmer’s Reference Manual.

The call completes after the specified timeout or when a message is available on one of the specified channels.

rtr_receive_message(&channel, RTR_NO_FLAGS, RTR_ANYCHAN, MsgBuffer, DataLen,
1000, &MsgStatusBlock);

An application can use a signaled receive to be alerted by RTR when a message is received. The application establishes a signal handler using the rtr_set_wakeup call, informing RTR where to call it back when the message is ready.

rtr_status_t
rtr_set_wakeup(
  procedure  )
void
wakeup_handler(void){
 rtr_receive_message();
}

main(){
 rtr_set_wakeup(wakeup_handler);
 sleep();
}

When using rtr_set_wakeup in a multithreaded application, be careful not to call any non-reentrant functions or tie up system resources unnecessarily inside the callback routine.

The rtr_open_channel parameters are further described in the VSI Reliable Transaction Router C Application Programmer’s Reference Manual .

Use the rtr_start_tx call when the application must set a client-side transaction timeout to ensure that both client and server do not wait too long for a message. When a transaction is started with rtr_send_to_server, no timeout is specified.

rtr_start_tx(&Channel,
        RTR_NO_FLAGS,
        RTR_NO_TIMOUTMS,
        RTR_NO_JOINCHAN); //or NULL

The two-phase commit process includes prepare and commit phases. A transaction is tentatively accepted or rejected during the prepare phase.

To initiate the prepare phase, the server application specifies the RTR_F_OPE_EXPLICIT_PREPARE flag when opening the channel, and can use the message rtr_mt_prepare to check commit status. The message indicates to the server application that it is time to prepare any updates for a later commit or rollback operation. RTR lets the server application explicitly accept a transaction using the RTR_F_OPE_EXPLICIT_ACCEPT flag on the rtr_open_channel call. Alternatively, RTR implicitly accepts the transaction after receiving the rtr_mt_accepted message when the server issues its next rtr_receive_message call.

The commit process is initiated by the client application when it issues a call to RTR indicating that the client "accepts" the transaction. This does not mean that the transaction is fully accepted, only that the client is prepared to accept it. RTR then asks the server applications participating in the transaction if they are prepared to accept the transaction. A server application that is prepared to accept the transaction votes its intention by issuing the rtr_accept_tx call, an "accept" vote. A server application that is not prepared to accept the transaction issues the rtr_reject_tx call, a "not accept" vote. Issuing all votes concludes the prepare phase.

When RTR has collected all votes from all participating server applications, it determines if the transaction is to be committed. If all collected votes are "accept," the transaction is committed; RTR informs all participating channels. If any vote is "not accept," the transaction is not committed. A server application can expose the prepare phase of two-phase commit by using the rtr_mt_prepare message type with the RTR_F_OPE_EXPLICIT_PREPARE flag. If the application's rtr_open_channel call sets neither the RTR_F_OPE_EXPLICIT_ACCEPT nor RTR_F_OPE_EXPLICIT_PREPARE flag, both prepare and accept processing are implicit.

Your application can use the level of participation that makes the most sense for your business and operations needs.

A participant can reject the transaction up to the time RTR has sent the rtr_mt_prepare message type to the server in the rtr_accept_tx call. A participant can reject the transaction up to the time the rtr_accept_tx call is executed. Once the client application has called rtr_accept_tx, the result cannot be changed.

rtr_start_tx

rtr_send_to_server

rtr_accept_tx

rtr_receive_message

In certain database applications, where the database manager does not let an application explicitly prepare the database, transactions can simply be accepted or rejected. Server applications that do not specify the RTR_F_EXPLICIT_ACCEPT flag in their rtr_open_channel call implicitly accept the in-progress transaction when an rtr_receive_message call is issued after the last message has been received for the transaction. This call returns the final status for the transaction, rtr_mt_accepted or rtr_mt_rejected. If neither the RTR_F_OPE_EXPLICIT_ACCEPT nor the RTR_F_OPE_EXPLICT_PREPARE flags are set in the rtr_open_channel call, then both prepare and accept processing will be implicit.

For server optimization, the server can signal its acceptance of a transaction with either rtr_reply_to_client, using the RTR_F_REP_ACCEPT flag, or with the client issuing the rtr_send_to_server call, using the RTR_F_SEN_ACCEPT flag. This helps to minimize network traffic for transactions by increasing the likelihood that the data message and the RTR accept message will be sent in the same network packet.

If a failure occurs after the SQL commit but before receipt of a message starting the next transaction, RTR does not know the difference.

If a failure occurs after an rtr_receive_message call is made to begin a new transaction, RTR assumes a successful commit if a server calls rtr_receive_message after receiving the rtr_mt_accepted message and will forget the transaction. There is no replay following these events.

Under certain conditions, transactions may become blocked or hung, and applications can use certain RTR features to clear such roadblocks. Often, a blocked state can be cleared by the system manager using the SET TRANSACTION CLI command to change the state of a transaction, for example, to DONE. Only certain states, however, can be changed, and changing the state of a transaction manually can endanger the integrity of the application database. For information on using the SET TRANSACTION command from the CLI, see the VSI Reliable Transaction Router System Manager’s Manual.

rtr_tid_t tid;
rtr_channel_t pchannel;
rtr_qualifier_value_t select qualifiers[4];
rtr_qualifier_value_t set_qualifiers[3];
int select_idx = 0;
int set_idx = 0;

rtr_get_tid(pchannel, RTR_F_TID_RTR, &tid);

/* Transaction branch to get current state */
 select_qualifiers[select_idx].qv_qualifier = rtr_txn_state;
 select_qualifiers[select_idx].qv_value = &rtr_txn_state_commit;
 select _ idx++;
/* Transaction branch to set new state */
 set_qualifiers[set_idx].qv_qualifier = rtr_txn_state;
 set_qualifiers[set_idx].qv_value = &rtr_txn_state_exception;
 set_idx++;

sts = rtr_set_info(pchannel,
     (rtr_flags_t) 0,
     (rtr_verb_t)verb_set,
     rtr_transaction_object,
     select_qualifiers,
     set_qualifiers);
if(sts != RTR_STS_OK)
       write an error;

sts = rtr_receive_message(
  /* channel  */ &channel_id,
  /* flags    */ RTR_NO_FLAGS,
  /* prcvchan */ pchannel,
  /* pmsg     */ msg,
  /* maxlen   */ RTR_MAX_MSGLEN,
  /* timoutms */ 0,
  /* pmsgsb   */ &msgsb);
if (sts == RTR_STS_OK){
 const rtr_status_data_t *pstatus = (rtr_status_data_t *)msg;
 rtr_uns_32_t num;

 switch(pstatus -> status)
 {
 case RTR_SETTRANDONE: /*Set Tran done successfully */
 memcpy(&num,(char)msg+sizeof(rtr_status_data_t),
  sizeof(rtr_uns_32_t));

 printf(" %d transaction(s) have been processed\n");
 break;
default;
}
}

RTR keeps track of how many times a transaction is presented to a server application before it is VOTED. The rule is: three strikes and you're out! After the third strike, RTR rejects the transaction with the reason RTR_STS_SRVDIED. The server application has committed the transaction and the client believes that the transaction is committed. The transaction is flagged as an exception and the database is not committed. Such an exception transaction can be manually committed if necessary. This process eliminates the possibility that a single rogue transaction can crash all available copies of a server application at both primary and secondary sites.

Application design can change this behavior. The application can specify the retry count to use when in recovery using the /recovery_retry_count qualifier in the rtr_set_info call, or the system administrator can set the retry count from the RTR CLI with the SET PARTITION command. If no recovery retry count is specified, RTR retries replay three times. For recovery, retries are infinite. For more information on the SET PARTITION command, refer to the VSI Reliable Transaction Router System Manager’s Manual; for more information on the rtr_set_info call, refer to the VSI Reliable Transaction Router C Application Programmer’s Reference Manual.

When a node is down, the operator can select a different backend to use for the server restart. To complete any incomplete transactions, RTR searches the journals of all backend nodes of a facility for any transactions for the key range specified in the server's rtr_open_channel call.

An application enables a callout server by setting a flag on the rtr_open_channel call.

  rtr_ope_flag_t
          flags=RTR_F_OPE_SERVER |RTR_F_OPE_TR_CALL_OUT