High Availability Guide : Watchdog sample : Failure situations and Watchdog actions : Communication links between the Watchdog and Primary, and between the Primary and Secondary, are down Scenario
  
Communication links between the Watchdog and Primary, and between the Primary and Secondary, are down Scenario
The connections between the Watchdog and the Primary server, and between the Primary server and Secondary server, are broken.
Remedy
For the Watchdog to continue monitoring the Primary server, switch the Secondary server to be the new Primary and set this new Primary to the PRIMARY ALONE state. Later, set up a new Secondary server and synchronize it with the Primary.
1 Server1's role is Primary Uncertain. However, from the watchdog's point of view, Server1 is unavailable, not Primary Uncertain.
Watchdog instructs Server2:
HSB SET PRIMARY ALONE
Applications switch from Server1 to Server 2.
2 Both servers believe they are primary. If a program or an administrator manual intervention) switches Server1 from Primary Uncertain to Primary Alone, then there are two active primaries, both of which could be updating data, and the differences would not be resolvable.
After network connections are fixed, Watchdog instructs Server 1:
HSB SWITCH SECONDARY
Watchdog instructs Server2:
HSB NETCOPY
HSB CONNECT
3 If the transaction log fills up, you may have to switch the PRIMARY ALONE server to STANDALONE. In that case, you will also need to execute HSB NETCOPY before you reconnect the servers. If the transaction log does not fill
up, then you must skip the NETCOPY command.
Symptoms
The Watchdog poll fails at the Primary server. The Secondary server and Primary server have lost their connections to each other; therefore Server2 is in the state SECONDARY ALONE, and the Primary (if it can be contacted) will report that its state is PRIMARY UNCERTAIN or PRIMARY ALONE.
The beginning of this scenario assumes that applications are possibly connected to the old Primary. However, since the old Primary is in the PRIMARY UNCERTAIN state, the applications are unable to perform updates. Note that it is also possible that the applications connected to Server1 may have lost their communication link and no longer know that the old Primary exists.
How to recover when communication links between the Watchdog and Primary, and between the Primary and Secondary, are down
To recover from the scenario where the connections between the Watchdog and the Primary server, and between the Primary server and Secondary server, are broken, perform the steps necessary to make the hot standby server (the Secondary server) replace the Primary.
About this task
To allow the Secondary server to replace the Primary, do the following:
Procedure
1 If the old Primary is in the PRIMARY UNCERTAIN state or is cut off from the applications as well as the Secondary, then set the Secondary server to PRIMARY ALONE state using the command:
ADMIN COMMAND 'hotstandby set primary alone';
2 Reconnect applications to the new Primary.
3 Fix the network or the broken connections to the old Primary.
4 Check the server states. Both servers must now be running.
5 If the new Primary is in STANDALONE state (for example, because the new Primary's transaction log filled up while the connections were being fixed):
a Set the new primary to PRIMARY ALONE state using the command:
ADMIN COMMAND 'hotstandby set primary alone';
b Copy the database from the new Primary to the new Secondary. Read Synchronizing primary and secondary servers for details.
6 If the new Primary is in PRIMARY ALONE state:
Switch the old Primary to be the new Secondary server using the command:
ADMIN COMMAND 'hotstandby switch secondary';
7 Reconnect the new Primary to the new Secondary using the command:
ADMIN COMMAND 'hotstandby connect';
Further scenarios where communication links between the Watchdog and Primary, and between the Primary and Secondary, are down
If an application receives error message 10047 or 14537 from the new Primary:
Try to connect to the old Secondary to check if it has switched to be the new Primary.
If the old Secondary is not switched to be the new Primary, re-execute the transaction with the original Primary in PRIMARY ALONE state.
See also
Failure situations and Watchdog actions