Monday, January 7, 2013

Rational Team Concert - Manual failover techniques

Deploy each of the jts.war, ccm.war, admin.war on their respective Primary and Backup servers so that you can use idle standby as a strategy for failover in high availability environments.

The idle standby configuration enables recovery from failover to help ensure minimal impact on business operations during planned or unplanned server outages.

This post covers implementation of the idle standby configuration with Jazz™ Team Server using WebSphere® Application Server.

Important: Jazz Team Server applications allow only a single server to be active at any one time to a repository; therefore, the backup (or Idle) server is configured to never run asynchronous (or background) tasks. If a switch is made to the backup server, you must plan to bring the primary server back up as quickly as possible.

The topology diagram illustrates the configuration for basic high availability when using idle standby. In the following figure, the IBM® HTTP Server is used to direct incoming traffic to one of the two WebSphere Application Servers, Primary Server A or Backup Server B.


Note: For both cold standby or idle standby, you must ensure that the Primary server is completely stopped prior to failover. Also be sure that there are no running processes that are interacting with the database.

For the example:
Server:
- IBM HTTP Server 7.0 Fix Pack 23
Primary Server A:
- WebSphere Application Server 7.0 Fix Pack 23
Backup Server B:
- WebSphere Application Server 7.0 Fix Pack 23
 

Installing and configuring a Jazz application on primary and backup servers

To install and configure copies of the RTC applications on the primary and backup servers, see Setting up a WebSphere® Application Server and Deploying the Rational solution for Collaborative Lifecycle Management applications on WebSphere Application Server.

For the example:
Primary Server A: server1
Backup Server B: server2


- After deploying the web applications, configure the primary server by following the steps: in Running the setup wizard (Custom setup).
Important: Do not run the Jazz Team Server setup wizard on the backup server. It must be run only on the primary server.
- Set the public URL to point to the IBM HTTP Server and not the application server.
- After installing and configuring the primary server, shut it down, and copy its configuration files into the backup server installation.

Configuring high availability for both primary and backup servers

Examples in this post are for the jts.war application. You can use the same procedure for the ccm.war and admin.war applications.

The jts.war application is typically installed with a single application server as its target. With the introduction of the Web server, the application container configuration must be modified to allow routing through the Web server. 

To modify the application:

  1. In the WebSphere Console, click the jts.war application link under Enterprise Applications.
  2. Select Manage Modules.
  3. Select the check box for the jts.war application module.
  4. In the list of clusters and servers, choose both the Web server and application server, and then click Apply.
  5. Click OK, and then click Save changes.
  6. Restart the jts.war application.
Reconfigure both the primary and backup Jazz™ Team Server instances to reference the same location for the full text index.
To keep the index up to date and available to both the primary and back up servers, update the com.ibm.team.fulltext.indexLocation and com.ibm.team.jfs.index.root.directory properties in the teamserver.properties files on both the primary and backup servers to store the index on a shared drive.

This property setting is an example of what you can see on Linux:
For JTS:
com.ibm.team.fulltext.indexLocation=/opt/FACTORY/rtc/conf/jts/sharedIndex/workitemindex
com.ibm.team.jfs.index.root.directory=sharedIndex

For CCM:
com.ibm.team.fulltext.indexLocation=/opt/FACTORY/rtc/conf/ccm/sharedIndex/workitemindex
com.ibm.team.jfs.index.root.directory=sharedIndex

Turning off asynchronous tasks on the backup server

To avoid any possible data contention between the two running Jazz Team Servers, asynchronous tasks must be turned off on the backup server.

Add the following line to the teamserver.properties file on the Backup server:
com.ibm.team.repository.scheduler.migration.mode.enabled=true

Restart the jts.war application on the backup server.

Note: Ensure that the preceding property is preserved on the backup server after any updates are applied to the asynchronous task that changes the configuration file on the primary server.

Note: When asynchronous tasks are turned off, the backup server will not run the asynchronous tasks. If circumstances dictate that the backup server is used as the primary server for an extended period of time, this property should be reset to false to enable the background tasks in the Advanced Properties. Make sure to set it to true before restarting the primary server.

Editing the Web server plugin_cfg.xml file for idle standby

Each time a WebSphere Application Server is configured to route requests through a Web server to an application server, the Web server plugin.xml file is updated with the connection information for that application server. Replace and then edit the following section of the plugin-cfg.xml file on the Web server to complete the configuration.
This plugin-cfg.xml file is located in the plugin\config\webserver1 folder of the Web server, where webserver1 is the name that you assigned.

Synchronizing the configuration files for the primary and backup servers

Customized server properties are not automatically synchronized between the primary and backup servers. In order to ensure that the backup servers have consistent configuration attributes, all customized settings must be propagated to the backup servers before they become active.
Periodically backing up from the primary server and restoring those files to the backup server are sufficient to keep customizable server properties in sync.

Note: If there are any configuration changes that require that you restart the primary server, then the backup server must also be restarted to get these changes.

This list includes configuration files that might need to be synchronized between the primary and backup servers.

 Jazz Team Server
        <JazzInstallDir>/server/conf/jts/teamserver.properties
        <JazzInstallDir>/server/conf/jts/log4j.properties

 CCM
        <JazzInstallDir>/server/conf/ccm/teamserver.properties
        <JazzInstallDir>/server/conf/ccm/log4j.properties

  ADMIN
        <JazzInstallDir>/server/conf/admin/friends.rtf
        <JazzInstallDir>/server/conf/admin/log4j.properties
        <JazzInstallDir>/server/conf/admin/admin.properties


Verifying the server setup for manual failover ability

To verify the manual failover ability of the WebSphere Application Server, stop the primary server and start the backup server.




Login to see the status summary in JTS:




Detecting failure on the primary server

Important: You must ensure that the primary server is completely stopped before switching over to the failover server. Also be sure that there are no running processes that are interacting with the database.

To achieve high availability, you must know when the primary server is down. This knowledge is especially important for this basic high-availability solution, which does not support automatic failover of the primary server to the backup server.

The process of detecting a failed server is a critical and time-sensitive task. Several factors can indicate that a server has failed, such as network problems, configuration problems, application overloading, or user error. Whatever solution you choose to detect server failures, you must ensure that the alert is as rapid as possible.

Additional information

All documentation contained in the post, is based on the online help on Installing the Rational solution for CLM (Manual failover techniques).