FATAL: requested timeline 3 is not a child of this server's history

When we perform a switch-over, the replica fails to start with below errors.

[root@oracleasm1 /u01]#  service postgresql-9.6 start

Starting postgresql-9.6 service:                           [FAILED]

Excerpt from Log :

cp: cannot stat `pg_xlog/RECOVERYXLOG': No such file or directory

cp: cannot stat `pg_xlog/RECOVERYXLOG': No such file or directory

user=,db=,9262FATAL:  requested timeline 3 is not a child of this server's history

user=,db=,9262DETAIL:  Latest checkpoint is at 0/6A000024 on timeline 2, but in the history of the requested timeline, the server forked off from that timeline at 0/690000C0.

user=,db=,9259LOG:  startup process (PID 9262) exited with exit code 1

Why this happens?

When you recover a slave to a new master it completes "recovery" and starts its own timeline. From this point on, it is now the master and cannot go back to being the slave without a rebuild.

What needs to be done?

A new Standby has to be built from scratch, if the database size is huge then a longer time to rebuild, and for this period newly promoted Primary will be running without Standby. pg_rewind can be used to turn the former master into a standby of the new master much more quickly than a complete rebuild.

There's also another issue like when Switch-over happens, Primary does clean shutdown, Wal-sender process sends all outstanding WAL records to the standby but it doesn't wait for them to be replicated before it exists. Here, Wal-receiver fails to apply those outstanding WAL records as it detects closure of connection and exits.

cp: cannot stat `pg_xlog/RECOVERYHISTORY': No such file or directory

cp: cannot stat `pg_xlog/RECOVERYXLOG': No such file or directory

user=,db=,14121FATAL:  requested timeline 3 does not contain minimum recovery point 0/6A00008C on timeline 2

user=,db=,14119LOG:  startup process (PID 14121) exited with exit code 1

user=,db=,14119LOG:  aborting startup due to startup process failure

user=,db=,14119LOG:  database system is shut down



Stay Tuned... 

Read more 

Configure Streaming Replication in PostgreSQL

All about Physical Replication and Log shipping in Postgres 

Streaming-Replication Sync and Async, benefits of streaming replication over Log-shipping