Replication

It is important to read at least chapter How storeBackup's replication works which gives you an explanation of the basic concepts and configurations.

To understand how replication works in detail, you have to know that it is an extension to lateLinks.

The internals of replication are explained with an example. First, create a backup with one replication:

$ mkdir /tmp/a
$ cd /tmp/a
$ mkdir -p source backup/default repl delta
$ cp /bin/ls source
$ cp /bin/ls source/ls1
$ storeBackupReplicationWizard.pl -m backup -c repl -d delta -S default
WARNING 2013.08.17 10:12:44 25548 cannot find any existing backup at master backup directory </tmp/a/backup>
INFO    2013.08.17 10:12:44 25548 1 series chosen:
INFO    2013.08.17 10:12:44 25548       <default>
INFO    2013.08.17 10:12:44 25548 wrote master backup configuration file </tmp/a/backup/storeBackupBaseTree.conf>
INFO    2013.08.17 10:12:44 25548 wrote backup copy configuration file </tmp/a/repl/storeBackupBaseTree.conf>
INFO    2013.08.17 10:12:44 25548 wrote delta cache configuration file </tmp/a/delta/deltaCache.conf>

Let's see what storeBackupReplicationWizard.pl generated:

$ cat backup/storeBackupBaseTree.conf | egrep -v '^\s*$|^#'
backupTreeName='Master Backup'
backupType=master
seriesToDistribute='default'
deltaCache=/tmp/a/delta

$ cat delta/deltaCache.conf | egrep -v '^\s*$|^#'
backupCopy0='Backup Copy' 'default'
;backupCopy1=
;backupCopy2=
;backupCopy3=
;backupCopy4=
;backupCopy5=
;backupCopy6=
;backupCopy7=
;backupCopy8=
;backupCopy9=

$ cat repl/storeBackupBaseTree.conf | egrep -v '^\s*$|^#'
backupTreeName='Backup Copy'
backupType=copy
seriesToDistribute='default'
deltaCache=/tmp/a/delta

Files available in all directories are:

$ find . -print | sort
.
./backup
./backup/default
./backup/storeBackupBaseTree.conf
./delta
./delta/deltaCache.conf
./repl
./repl/storeBackupBaseTree.conf
./source
./source/ls
./source/ls1

Now, the first backup with lateLinks:

$ storeBackup.pl -s source -b backup --lateLinks
.....
$ find . -print | sort
.
./backup
./backup/default
./backup/default/2013.08.17_10.13.41
./backup/default/2013.08.17_10.13.41/ls1.bz2
./backup/default/2013.08.17_10.13.41/.md5CheckSums.bz2
./backup/default/2013.08.17_10.13.41/.md5CheckSums.info
./backup/default/2013.08.17_10.13.41/.storeBackupLinks
./backup/default/2013.08.17_10.13.41/.storeBackupLinks/linkFile.bz2
./backup/storeBackupBaseTree.conf
./delta
./delta/deltaCache.conf
./repl
./repl/storeBackupBaseTree.conf
./source
./source/ls
./source/ls1

Nothing new up to now. A compressed copy of ls1 from source can be found in backup now. The file ls is missing because of deduplication. Also, the meta data is available in backup plus the linkFile (see previous chapter).

To finalize the backup, run storeBackupUpdateBackup.pl:

$ storeBackupUpdateBackup.pl -b backup
BEGIN     2013.08.17 10:14:59 25914 checking references and backup copying in <backup>
VERSION   2013.08.17 10:14:59 25914 storeBackupUpdateBackup.pl, 3.4 +
INFO      2013.08.17 10:14:59 25914 creating lock file </tmp/storeBackup.lock>
INFO      2013.08.17 10:14:59 25914 reading <backup/storeBackupBaseTree.conf>
INFO      2013.08.17 10:14:59 25914 master backup: checking <Master Backup/default>
INFO      2013.08.17 10:14:59 25914     copying <2013.08.17_10.13.41> to </tmp/a/delta/default>
INFO      2013.08.17 10:14:59 25914 scanning directory <backup> for existing backups
INFO      2013.08.17 10:14:59 25914 scanning directory <backup/default> for existing backups
STATISTIC 2013.08.17 10:14:59 25914 found 1 backup series, 1 backups, 0 renamed backups
INFO      2013.08.17 10:14:59 25914 consistency check finished successfully
INFO      2013.08.17 10:14:59 25914 found no references to backups from lateLinks that need
                                    storeBackupUpdateBackup run
INFO      2013.08.17 10:14:59 25914 (1/0) updating </tmp/a/backup/default/2013.08.17_10.13.41>
INFO      2013.08.17 10:14:59 25914 phase 1: mkdir, symlink and compressing files
STATISTIC 2013.08.17 10:14:59 25914 created 0 directories
STATISTIC 2013.08.17 10:14:59 25914 created 0 symbolic links
STATISTIC 2013.08.17 10:14:59 25914 compressed 0 files
STATISTIC 2013.08.17 10:14:59 25914 used 0.0  instead of 0.0  (0 <- 0 ; 0.0%)
INFO      2013.08.17 10:14:59 25914 phase 2: setting hard links
STATISTIC 2013.08.17 10:14:59 25914 linked 1 files
INFO      2013.08.17 10:14:59 25914 phase 3: setting file permissions
STATISTIC 2013.08.17 10:14:59 25914 set permissions for 2 files
INFO      2013.08.17 10:14:59 25914 phase 4: setting directory permissions
STATISTIC 2013.08.17 10:14:59 25914 set permissions for 0 directories
INFO      2013.08.17 10:14:59 25914 reading <backup/storeBackupBaseTree.conf>
INFO      2013.08.17 10:14:59 25914 scanning directory <backup> for existing backups
INFO      2013.08.17 10:14:59 25914 scanning directory <backup/default> for existing backups
STATISTIC 2013.08.17 10:14:59 25914 found 1 backup series, 1 backups, 0 renamed backups
INFO      2013.08.17 10:14:59 25914 consistency check finished successfully
INFO      2013.08.17 10:14:59 25914 found no references to backups from lateLinks that need
                                    storeBackupUpdateBackup run
INFO      2013.08.17 10:14:59 25914 deleting in deltaCache </tmp/a/delta> processedBackups
INFO      2013.08.17 10:14:59 25914     age for deletion is > 99d (delete backups older than
                                        Fri 2013.05.10 10:14:59)
STATISTIC 2013.08.17 10:14:59 25914                       duration = 1s
STATISTIC 2013.08.17 10:14:59 25914  [sec] |      user|    system
STATISTIC 2013.08.17 10:14:59 25914 -------+----------+----------
STATISTIC 2013.08.17 10:14:59 25914 process|      0.07|      0.03
STATISTIC 2013.08.17 10:14:59 25914 childs |      0.05|      0.00
STATISTIC 2013.08.17 10:14:59 25914 -------+----------+----------
STATISTIC 2013.08.17 10:14:59 25914 sum    |      0.12|      0.03 => 0.15
INFO      2013.08.17 10:14:59 25914 syncing ...
END       2013.08.17 10:14:59 25914 checking references and copying in <backup>

$ find . -print | sort
.
./backup
./backup/default
./backup/default/2013.08.17_10.13.41
./backup/default/2013.08.17_10.13.41/ls1.bz2
./backup/default/2013.08.17_10.13.41/ls.bz2
./backup/default/2013.08.17_10.13.41/.md5CheckSums.bz2
./backup/default/2013.08.17_10.13.41/.md5CheckSums.info
./backup/default/2013.08.17_10.13.41/.storeBackupLinks
./backup/storeBackupBaseTree.conf
./delta
./delta/default
./delta/default/2013.08.17_10.13.41
./delta/default/2013.08.17_10.13.41/ls1.bz2
./delta/default/2013.08.17_10.13.41/.md5CheckSums.bz2
./delta/default/2013.08.17_10.13.41/.md5CheckSums.info
./delta/default/2013.08.17_10.13.41/.storeBackupLinks
./delta/default/2013.08.17_10.13.41/.storeBackupLinks/linkFile.bz2
./delta/deltaCache.conf
./delta/processedBackups
./repl
./repl/storeBackupBaseTree.conf
./source
./source/ls
./source/ls1

Lots of things have changed now:

The master backup in backup is completed. In the backup directory, the files ls and ls1 have been hard linked and the command file linkFile is away.
In the beginning of the output of storeBackupUpdateBackup.pl you can see the following lines:
```
 reading <backup/storeBackupBaseTree.conf>
 master backup: checking <Master Backup/default>
 copying <2013.08.17_10.13.41> to </tmp/a/delta/default>
```
The program detected that that lateLinks backup was not yet copied into the delta cache. Before completing the backup, these deltas were copied into the delta cache. Have a look at the output of the find command to see that copy in directory delta. The delta cache collects all new backups (to be replicated) from the master backup (backup) until they are transferred to the final replication directory (repl) and also stores them later for a while.
You also see, that the directory processedBackups was created in delta.

Now, run storeBackupUpdateBackup.pl a second time, but on repl:

$ storeBackupUpdateBackup.pl -b repl
BEGIN     2013.08.17 10:33:03 28608 checking references and backup copying in <repl>
VERSION   2013.08.17 10:33:03 28608 storeBackupUpdateBackup.pl, 3.4 +
INFO      2013.08.17 10:33:03 28608 creating lock file </tmp/storeBackup.lock>
INFO      2013.08.17 10:33:03 28608 reading <repl/storeBackupBaseTree.conf>
INFO      2013.08.17 10:33:03 28608 reading </tmp/a/delta/deltaCache.conf>
INFO      2013.08.17 10:33:03 28608 copying </tmp/a/delta/default/2013.08.17_10.13.41> to <repl/default>
INFO      2013.08.17 10:33:03 28608 scanning directory <repl> for existing backups
INFO      2013.08.17 10:33:03 28608 scanning directory <repl/default> for existing backups
STATISTIC 2013.08.17 10:33:03 28608 found 1 backup series, 1 backups, 0 renamed backups
INFO      2013.08.17 10:33:03 28608 consistency check finished successfully
INFO      2013.08.17 10:33:03 28608 found no references to backups from lateLinks that need
                                    storeBackupUpdateBackup run
INFO      2013.08.17 10:33:03 28608 (1/0) updating </tmp/a/repl/default/2013.08.17_10.13.41>
INFO      2013.08.17 10:33:03 28608 phase 1: mkdir, symlink and compressing files
STATISTIC 2013.08.17 10:33:03 28608 created 0 directories
STATISTIC 2013.08.17 10:33:03 28608 created 0 symbolic links
STATISTIC 2013.08.17 10:33:03 28608 compressed 0 files
STATISTIC 2013.08.17 10:33:03 28608 used 0.0  instead of 0.0  (0 <- 0 ; 0.0%)
INFO      2013.08.17 10:33:03 28608 phase 2: setting hard links
STATISTIC 2013.08.17 10:33:03 28608 linked 1 files
INFO      2013.08.17 10:33:03 28608 phase 3: setting file permissions
STATISTIC 2013.08.17 10:33:03 28608 set permissions for 2 files
INFO      2013.08.17 10:33:03 28608 phase 4: setting directory permissions
STATISTIC 2013.08.17 10:33:03 28608 set permissions for 0 directories
INFO      2013.08.17 10:33:03 28608 reading <repl/storeBackupBaseTree.conf>
INFO      2013.08.17 10:33:03 28608 marked <default/2013.08.17_10.13.41> as linked in </tmp/a/delta>
INFO      2013.08.17 10:33:03 28608 scanning directory <repl> for existing backups
INFO      2013.08.17 10:33:03 28608 scanning directory <repl/default> for existing backups
STATISTIC 2013.08.17 10:33:03 28608 found 1 backup series, 1 backups, 0 renamed backups
INFO      2013.08.17 10:33:03 28608 consistency check finished successfully
INFO      2013.08.17 10:33:03 28608 found no references to backups from lateLinks that need
                                    storeBackupUpdateBackup run
INFO      2013.08.17 10:33:03 28608 backup </tmp/a/delta/default/2013.08.17_10.13.41> copied to <Backup Copy>
INFO      2013.08.17 10:33:03 28608     moving backup to </tmp/a/delta/processedBackups/default>
INFO      2013.08.17 10:33:03 28608 deleting in deltaCache </tmp/a/delta> processedBackups
INFO      2013.08.17 10:33:03 28608     age for deletion is > 99d (delete backups older than
                                        Fri 2013.05.10 10:33:03)
INFO      2013.08.17 10:33:03 28608 checking series <default>
INFO      2013.08.17 10:33:03 28608     default -> 2013.08.17_10.13.41 - not old enough to delete
STATISTIC 2013.08.17 10:33:03 28608                       duration = 1s
STATISTIC 2013.08.17 10:33:03 28608  [sec] |      user|    system
STATISTIC 2013.08.17 10:33:03 28608 -------+----------+----------
STATISTIC 2013.08.17 10:33:03 28608 process|      0.09|      0.01
STATISTIC 2013.08.17 10:33:03 28608 childs |      0.05|      0.00
STATISTIC 2013.08.17 10:33:03 28608 -------+----------+----------
STATISTIC 2013.08.17 10:33:03 28608 sum    |      0.14|      0.01 => 0.15
INFO      2013.08.17 10:33:03 28608 syncing ...
END       2013.08.17 10:33:03 28608 checking references and copying in <repl>

$ find . -print | sort
.
./backup
./backup/default
./backup/default/2013.08.17_10.13.41
./backup/default/2013.08.17_10.13.41/ls1.bz2
./backup/default/2013.08.17_10.13.41/ls.bz2
./backup/default/2013.08.17_10.13.41/.md5CheckSums.bz2
./backup/default/2013.08.17_10.13.41/.md5CheckSums.info
./backup/default/2013.08.17_10.13.41/.storeBackupLinks
./backup/storeBackupBaseTree.conf
./delta
./delta/default
./delta/default/2013.08.17_10.13.41.copied
./delta/default/2013.08.17_10.13.41.linked
./delta/deltaCache.conf
./delta/processedBackups
./delta/processedBackups/default
./delta/processedBackups/default/2013.08.17_10.13.41
./delta/processedBackups/default/2013.08.17_10.13.41/ls1.bz2
./delta/processedBackups/default/2013.08.17_10.13.41/.md5CheckSums.bz2
./delta/processedBackups/default/2013.08.17_10.13.41/.md5CheckSums.info
./delta/processedBackups/default/2013.08.17_10.13.41/.storeBackupLinks
./delta/processedBackups/default/2013.08.17_10.13.41/.storeBackupLinks/linkFile.bz2
./repl
./repl/default
./repl/default/2013.08.17_10.13.41
./repl/default/2013.08.17_10.13.41/ls1.bz2
./repl/default/2013.08.17_10.13.41/ls.bz2
./repl/default/2013.08.17_10.13.41/.md5CheckSums.bz2
./repl/default/2013.08.17_10.13.41/.md5CheckSums.info
./repl/default/2013.08.17_10.13.41/.storeBackupLinks
./repl/storeBackupBaseTree.conf
./source
./source/ls
./source/ls1

A look at the file list and the log of storeBackupUpdateBackup.pl shows:

The master backup (directory backup) has not changed.
The deltas have been copied to the replication directory (repl) where the copy of the backup should be stored. Additionally, this copy was used to complete the backup in repl. A diff between backup and repl shows
```
$ diff -r backup repl 
diff -r backup/storeBackupBaseTree.conf repl/storeBackupBaseTree.conf
21c21
< backupTreeName='Master Backup'
---
> backupTreeName='Backup Copy'
26c26
< backupType=master
---
> backupType=copy
```
That means that only the configuration file for the replication is different. The backups inside those directory are absolutely identical. You can e.g. run storeBackupDel.pl or storeBackupCheckBackup.pl on both of them.
The copy of the deltas in the delta cache directory was moved to its sub directory processedBackups. The status of that replication is stored in the files:
```
$ cat ./delta/default/2013.08.17_10.13.41.copied
Backup Copy

$ cat ./delta/default/2013.08.17_10.13.41.linked
Backup Copy
```
This means, that the backup from series default from 2013.08.17_10.13.41 has been copied and linked to backup copy Backup Copy. If you replicate to more than one other directory (e.g. on a usb disk) then you will see the name (like Backup Copy in this case) of that other backup copy in these files (.copied and .linked) also after it has been copied and linked. By this way, storeBackup can decide if a backup was fully copied and linked to all replication directories. After being sure about this, the deltas for that particular backup are moved to the sub directory processedBackups in the delta cache. (We will see later how to use this if somethings goes wrong in the replication process.) These processed backups are stored for some time you can define. The default of 99 days is very conservative and far too long.

With the example above, it is easy to see how the concept of replication is realized:⁵⁸

run storeBackup.pl -b backup and create the lateLinks based delta backup in directory backup.
run storeBackupUpdateBackup.pl -b backup and
1. copy the deltas from the backup to the delta cache (directory delta)
2. complete (make hard links etc.) in the backup in directory backup
run storeBackupUpdateBackup.pl -b repl and
1. copy the deltas from the delta cache (directory delta) to the replication target directory repl
2. write the name of the replication (Backup Copy) to file timestamp.copied
3. complete (make hard links etc.) the replication target / copy the directory repl (in principle the same as in step 2b described above).
4. move the just copied backup in the delta cache into the sub directory processedBackups and write the name of the replication (Backup Copy) to file timestamp.linked

With the knowledge of the fundamental principles of the replication it is easy to react if something goes wrong. It is always just moving, copying or adding lines to files. Let's simulate an interrupted replication. (New example from the beginning.)

First, create a new backup and replicate it:

$ mkdir /tmp/x
$ cd /tmp/x
$ mkdir -p source backup/default repl delta
$ cp /bin/ls source
$ storeBackupReplicationWizard.pl -m backup -c repl -d delta -S default
$ storeBackup.pl -s source -b backup --lateLinks
$ storeBackupUpdateBackup.pl -b backup
$ storeBackupUpdateBackup.pl -b repl

Then, add a file to source, backup and replicate again.

$ cp /bin/pwd source
$ storeBackup.pl -s source -b backup --lateLinks
$ storeBackupUpdateBackup.pl -b backup
$ storeBackupUpdateBackup.pl -b repl

I created the following backups:

$ ls -1 backup/default/
2013.08.18_09.45.16
2013.08.18_09.49.21

No, let's imagine, the second backup is broken, maybe because the run of storeBackupUpdateBackup.pl was interrupted, had write errors or for whatever reason. Because I wouldn't know the details of the damage and which files may be missing, I'm deleting the newly replicated backup:

$ ls -1 repl/default/
2013.08.18_09.45.16
2013.08.18_09.49.21
$ rm -r repl/default/2013.08.18_09.49.21

Re-running storeBackupUpdateBackup.pl on the replication (One backup is missing!) results in:

$ storeBackupUpdateBackup.pl -b repl
BEGIN     2013.08.18 09:57:46  6852 checking references and backup copying in <repl>
VERSION   2013.08.18 09:57:46  6852 storeBackupUpdateBackup.pl, 3.4 +
INFO      2013.08.18 09:57:46  6852 creating lock file </tmp/storeBackup.lock>
INFO      2013.08.18 09:57:46  6852 reading <repl/storeBackupBaseTree.conf>
INFO      2013.08.18 09:57:46  6852 reading </tmp/x/delta/deltaCache.conf>
INFO      2013.08.18 09:57:46  6852 scanning directory <repl> for existing backups
INFO      2013.08.18 09:57:46  6852 scanning directory <repl/default> for existing backups
STATISTIC 2013.08.18 09:57:46  6852 found 1 backup series, 1 backups, 0 renamed backups
INFO      2013.08.18 09:57:46  6852 consistency check finished successfully
INFO      2013.08.18 09:57:46  6852 found no references to backups from lateLinks that need
                                    storeBackupUpdateBackup run
INFO      2013.08.18 09:57:46  6852 everything is updated, nothing to do
INFO      2013.08.18 09:57:46  6852 deleting in deltaCache </tmp/x/delta> processedBackups
INFO      2013.08.18 09:57:46  6852     age for deletion is > 99d (delete backups older
                                        than Sat 2013.05.11 09:57:46)
INFO      2013.08.18 09:57:46  6852 checking series <default>
INFO      2013.08.18 09:57:46  6852     default -> 2013.08.18_09.45.16 - not old enough to delete
INFO      2013.08.18 09:57:46  6852     default -> 2013.08.18_09.49.21 - not old enough to delete
STATISTIC 2013.08.18 09:57:46  6852                       duration = 1s
STATISTIC 2013.08.18 09:57:46  6852  [sec] |      user|    system
STATISTIC 2013.08.18 09:57:46  6852 -------+----------+----------
STATISTIC 2013.08.18 09:57:46  6852 process|      0.08|      0.01
STATISTIC 2013.08.18 09:57:46  6852 childs |      0.05|      0.00
STATISTIC 2013.08.18 09:57:46  6852 -------+----------+----------
STATISTIC 2013.08.18 09:57:46  6852 sum    |      0.13|      0.01 => 0.14
INFO      2013.08.18 09:57:46  6852 syncing ...
END       2013.08.18 09:57:46  6852 checking references and copying in <repl>

$ find repl -print | sort
repl
repl/default
repl/default/2013.08.18_09.45.16
repl/default/2013.08.18_09.45.16/ls.bz2
repl/default/2013.08.18_09.45.16/.md5CheckSums.bz2
repl/default/2013.08.18_09.45.16/.md5CheckSums.info
repl/default/2013.08.18_09.45.16/.storeBackupLinks
repl/storeBackupBaseTree.conf

So, the result is - nothing. storeBackupUpdateBackup.pl thinks everything is fine because there's nothing to replicate anymore. (The control files indicate that everything ran perfectly.)

If this is not fixed, the replication will run into problems with the next backup. To make things clear, let's provoke this:

$ storeBackup.pl -s source -b backup --lateLinks
$ storeBackupUpdateBackup.pl -b backup
$ storeBackupUpdateBackup.pl -b repl
$ storeBackupUpdateBackup.pl -b repl
BEGIN     2013.08.18 10:06:06  7050 checking references and backup copying in <repl>
VERSION   2013.08.18 10:06:06  7050 storeBackupUpdateBackup.pl, 3.4 +
INFO      2013.08.18 10:06:06  7050 creating lock file </tmp/storeBackup.lock>
INFO      2013.08.18 10:06:06  7050 reading <repl/storeBackupBaseTree.conf>
INFO      2013.08.18 10:06:06  7050 reading </tmp/x/delta/deltaCache.conf>
INFO      2013.08.18 10:06:06  7050 copying </tmp/x/delta/default/2013.08.18_10.05.54> to <repl/default>
INFO      2013.08.18 10:06:07  7050 scanning directory <repl> for existing backups
INFO      2013.08.18 10:06:07  7050 scanning directory <repl/default> for existing backups
STATISTIC 2013.08.18 10:06:07  7050 found 1 backup series, 2 backups, 0 renamed backups
ERROR     2013.08.18 10:06:07  7050 FATAL ERROR: link <../2013.08.18_09.49.21> to non existing dir in
                                    </tmp/x/repl/default/2013.08.18_10.05.54/.storeBackupLinks/linkTo>
ERROR     2013.08.18 10:06:07  7050 found 1 inconsistencies, please repair and check again

The FATAL ERROR message found a missing backup, so there is no chance to create the hard links.

Looking to the files in the delta cache (delta) and the replication target directory (repl) shows:

$ find delta repl -print | sort
delta
delta/default
delta/default/2013.08.18_09.45.16.copied
delta/default/2013.08.18_09.45.16.linked
delta/default/2013.08.18_09.49.21.copied
delta/default/2013.08.18_09.49.21.linked
delta/default/2013.08.18_10.05.54
delta/default/2013.08.18_10.05.54.copied
delta/default/2013.08.18_10.05.54/.md5CheckSums.bz2
delta/default/2013.08.18_10.05.54/.md5CheckSums.info
delta/default/2013.08.18_10.05.54/.storeBackupLinks
delta/default/2013.08.18_10.05.54/.storeBackupLinks/linkFile.bz2
delta/default/2013.08.18_10.05.54/.storeBackupLinks/linkTo
delta/deltaCache.conf
delta/processedBackups
delta/processedBackups/default
delta/processedBackups/default/2013.08.18_09.45.16
delta/processedBackups/default/2013.08.18_09.45.16/ls.bz2
delta/processedBackups/default/2013.08.18_09.45.16/.md5CheckSums.bz2
delta/processedBackups/default/2013.08.18_09.45.16/.md5CheckSums.info
delta/processedBackups/default/2013.08.18_09.45.16/.storeBackupLinks
delta/processedBackups/default/2013.08.18_09.45.16/.storeBackupLinks/linkFile.bz2
delta/processedBackups/default/2013.08.18_09.49.21
delta/processedBackups/default/2013.08.18_09.49.21/.md5CheckSums.bz2
delta/processedBackups/default/2013.08.18_09.49.21/.md5CheckSums.info
delta/processedBackups/default/2013.08.18_09.49.21/pwd.bz2
delta/processedBackups/default/2013.08.18_09.49.21/.storeBackupLinks
delta/processedBackups/default/2013.08.18_09.49.21/.storeBackupLinks/linkFile.bz2
delta/processedBackups/default/2013.08.18_09.49.21/.storeBackupLinks/linkTo
repl
repl/default
repl/default/2013.08.18_09.45.16
repl/default/2013.08.18_09.45.16/ls.bz2
repl/default/2013.08.18_09.45.16/.md5CheckSums.bz2
repl/default/2013.08.18_09.45.16/.md5CheckSums.info
repl/default/2013.08.18_09.45.16/.storeBackupLinks
repl/default/2013.08.18_10.05.54
repl/default/2013.08.18_10.05.54/.md5CheckSums.bz2
repl/default/2013.08.18_10.05.54/.md5CheckSums.info
repl/default/2013.08.18_10.05.54/.storeBackupLinks
repl/default/2013.08.18_10.05.54/.storeBackupLinks/linkFile.bz2
repl/default/2013.08.18_10.05.54/.storeBackupLinks/linkTo
repl/storeBackupBaseTree.conf

We can see the following:

In repl
1. Backup repl/default/2013.08.18_09.45.16 has been copied and completed.
  (.storeBackupLinks/linkFile.bz2 doesn't exist any more)
2. Backup repl/default/2013.08.18_10.05.54 was copied but not completed.
  (.storeBackupLinks/linkFile.bz2 still exists)
In delta/default
1. The first backup, 2013.08.18_09.45.16 is marked as copied and linked which is fine, see 1a above.
2. The second backup, 2013.08.18_09.49.21 is marked as copied and linked also - this is wrong, because it is missing in repl. (I deleted it to simulate an issue.)
3. The third backup, 2013.08.18_10.05.54 is marked as copied, but not linked which is correct, see 1b above.

To solve the problem, I could copy the whole completed backups from backup to repl. But in real life, this would be very time consuming and you may lose data availabe in repl but not in backup because of different deletion schemes.

Another way to solve the problem is to use linkToDirs.pl to copy / hard link the missing backup from backup directly to the correct place in repl. See section linkToDirs.pl for detailed information.

The most ``elegant'' and fastest way is to change the delta cache according to the situation in repl:

The missing backup (see 2b above) is marked as ``copied'' and ``linked'', so I remove these to markers. See the first command below.
ATTENTION: This only works because in this example, only one replication is configured. If more replications are configured, you have to delete the name of the replication in those files. If you simply delete the files, you might probably harm other replications!
But the backup (missing in repl) is not at its place in delta/default. Fortunately, it is still available in delta/processedBackups/default. With the second command below I move it to the right location.

Finally, the output of find shows all files at the right location:

$ rm delta/default/2013.08.18_09.49.21.copied delta/default/2013.08.18_09.49.21.linked
$ mv delta/processedBackups/default/2013.08.18_09.49.21 delta/default
$ find delta -print | sort
delta
delta/default
delta/default/2013.08.18_09.45.16.copied
delta/default/2013.08.18_09.45.16.linked
delta/default/2013.08.18_09.49.21
delta/default/2013.08.18_09.49.21/.md5CheckSums.bz2
delta/default/2013.08.18_09.49.21/.md5CheckSums.info
delta/default/2013.08.18_09.49.21/pwd.bz2
delta/default/2013.08.18_09.49.21/.storeBackupLinks
delta/default/2013.08.18_09.49.21/.storeBackupLinks/linkFile.bz2
delta/default/2013.08.18_09.49.21/.storeBackupLinks/linkTo
delta/default/2013.08.18_10.05.54
delta/default/2013.08.18_10.05.54.copied
delta/default/2013.08.18_10.05.54/.md5CheckSums.bz2
delta/default/2013.08.18_10.05.54/.md5CheckSums.info
delta/default/2013.08.18_10.05.54/.storeBackupLinks
delta/default/2013.08.18_10.05.54/.storeBackupLinks/linkFile.bz2
delta/default/2013.08.18_10.05.54/.storeBackupLinks/linkTo
delta/deltaCache.conf
delta/processedBackups
delta/processedBackups/default
delta/processedBackups/default/2013.08.18_09.45.16
delta/processedBackups/default/2013.08.18_09.45.16/ls.bz2
delta/processedBackups/default/2013.08.18_09.45.16/.md5CheckSums.bz2
delta/processedBackups/default/2013.08.18_09.45.16/.md5CheckSums.info
delta/processedBackups/default/2013.08.18_09.45.16/.storeBackupLinks
delta/processedBackups/default/2013.08.18_09.45.16/.storeBackupLinks/linkFile.bz2

Running storeBackupUpdateBackup.pl on repl results in the following output:

$ storeBackupUpdateBackup.pl -b repl
BEGIN     2013.08.18 10:39:08  7958 checking references and backup copying in <repl>
VERSION   2013.08.18 10:39:08  7958 storeBackupUpdateBackup.pl, 3.4 +
INFO      2013.08.18 10:39:08  7958 removing old lock file of process <7050>
INFO      2013.08.18 10:39:08  7958 creating lock file </tmp/storeBackup.lock>
INFO      2013.08.18 10:39:08  7958 reading <repl/storeBackupBaseTree.conf>
INFO      2013.08.18 10:39:08  7958 reading </tmp/x/delta/deltaCache.conf>
INFO      2013.08.18 10:39:08  7958 copying </tmp/x/delta/default/2013.08.18_09.49.21> to <repl/default>
INFO      2013.08.18 10:39:08  7958 scanning directory <repl> for existing backups
INFO      2013.08.18 10:39:08  7958 scanning directory <repl/default> for existing backups
STATISTIC 2013.08.18 10:39:08  7958 found 1 backup series, 3 backups, 0 renamed backups
INFO      2013.08.18 10:39:08  7958     1 directory </tmp/x/repl/default/2013.08.18_09.45.16> has no linkFrom entry
INFO      2013.08.18 10:39:08  7958 autorepair: wrote linkFrom from </tmp/x/repl/default/2013.08.18_09.45.16> to
                                    </tmp/x/repl/default/2013.08.18_09.49.21>
INFO      2013.08.18 10:39:08  7958 
INFO      2013.08.18 10:39:08  7958 ----- repeating consistency check -----
INFO      2013.08.18 10:39:08  7958 scanning directory <repl> for existing backups
INFO      2013.08.18 10:39:08  7958 scanning directory <repl/default> for existing backups
STATISTIC 2013.08.18 10:39:08  7958 found 1 backup series, 3 backups, 0 renamed backups
INFO      2013.08.18 10:39:08  7958     1 directory </tmp/x/repl/default/2013.08.18_09.49.21> has no linkFrom entry
INFO      2013.08.18 10:39:08  7958 autorepair: wrote linkFrom from </tmp/x/repl/default/2013.08.18_09.49.21> to
                                    </tmp/x/repl/default/2013.08.18_10.05.54>
INFO      2013.08.18 10:39:08  7958 
INFO      2013.08.18 10:39:08  7958 ----- repeating consistency check -----
INFO      2013.08.18 10:39:08  7958 scanning directory <repl> for existing backups
INFO      2013.08.18 10:39:08  7958 scanning directory <repl/default> for existing backups
STATISTIC 2013.08.18 10:39:08  7958 found 1 backup series, 3 backups, 0 renamed backups
INFO      2013.08.18 10:39:08  7958 consistency check finished successfully
INFO      2013.08.18 10:39:08  7958 listing references:
INFO      2013.08.18 10:39:08  7958   /tmp/x/repl/default/2013.08.18_09.49.21
INFO      2013.08.18 10:39:08  7958     -> /tmp/x/repl/default/2013.08.18_09.45.16
INFO      2013.08.18 10:39:08  7958   /tmp/x/repl/default/2013.08.18_10.05.54
INFO      2013.08.18 10:39:08  7958     -> /tmp/x/repl/default/2013.08.18_09.49.21
INFO      2013.08.18 10:39:08  7958 (1/2) updating </tmp/x/repl/default/2013.08.18_09.49.21>
INFO      2013.08.18 10:39:08  7958 phase 1: mkdir, symlink and compressing files
STATISTIC 2013.08.18 10:39:08  7958 created 0 directories
STATISTIC 2013.08.18 10:39:08  7958 created 0 symbolic links
STATISTIC 2013.08.18 10:39:08  7958 compressed 0 files
STATISTIC 2013.08.18 10:39:08  7958 used 0.0  instead of 0.0  (0 <- 0 ; 0.0%)
INFO      2013.08.18 10:39:08  7958 phase 2: setting hard links
STATISTIC 2013.08.18 10:39:08  7958 linked 1 files
INFO      2013.08.18 10:39:08  7958 phase 3: setting file permissions
STATISTIC 2013.08.18 10:39:08  7958 set permissions for 2 files
INFO      2013.08.18 10:39:08  7958 phase 4: setting directory permissions
STATISTIC 2013.08.18 10:39:08  7958 set permissions for 0 directories
INFO      2013.08.18 10:39:08  7958 reading <repl/storeBackupBaseTree.conf>
INFO      2013.08.18 10:39:08  7958 marked <default/2013.08.18_09.49.21> as linked in </tmp/x/delta>
INFO      2013.08.18 10:39:08  7958 scanning directory <repl> for existing backups
INFO      2013.08.18 10:39:08  7958 scanning directory <repl/default> for existing backups
STATISTIC 2013.08.18 10:39:08  7958 found 1 backup series, 3 backups, 0 renamed backups
INFO      2013.08.18 10:39:08  7958 consistency check finished successfully
INFO      2013.08.18 10:39:08  7958 listing references:
INFO      2013.08.18 10:39:08  7958   /tmp/x/repl/default/2013.08.18_10.05.54
INFO      2013.08.18 10:39:08  7958     -> /tmp/x/repl/default/2013.08.18_09.49.21
INFO      2013.08.18 10:39:08  7958 (2/2) updating </tmp/x/repl/default/2013.08.18_10.05.54>
INFO      2013.08.18 10:39:08  7958 phase 1: mkdir, symlink and compressing files
STATISTIC 2013.08.18 10:39:08  7958 created 0 directories
STATISTIC 2013.08.18 10:39:08  7958 created 0 symbolic links
STATISTIC 2013.08.18 10:39:08  7958 compressed 0 files
STATISTIC 2013.08.18 10:39:08  7958 used 0.0  instead of 0.0  (0 <- 0 ; 0.0%)
INFO      2013.08.18 10:39:08  7958 phase 2: setting hard links
STATISTIC 2013.08.18 10:39:08  7958 linked 2 files
INFO      2013.08.18 10:39:08  7958 phase 3: setting file permissions
STATISTIC 2013.08.18 10:39:08  7958 set permissions for 2 files
INFO      2013.08.18 10:39:08  7958 phase 4: setting directory permissions
STATISTIC 2013.08.18 10:39:08  7958 set permissions for 0 directories
INFO      2013.08.18 10:39:08  7958 reading <repl/storeBackupBaseTree.conf>
INFO      2013.08.18 10:39:08  7958 marked <default/2013.08.18_10.05.54> as linked in </tmp/x/delta>
INFO      2013.08.18 10:39:08  7958 scanning directory <repl> for existing backups
INFO      2013.08.18 10:39:08  7958 scanning directory <repl/default> for existing backups
STATISTIC 2013.08.18 10:39:08  7958 found 1 backup series, 3 backups, 0 renamed backups
INFO      2013.08.18 10:39:08  7958 consistency check finished successfully
INFO      2013.08.18 10:39:08  7958 found no references to backups from lateLinks that need storeBackupUpdateBackup run
INFO      2013.08.18 10:39:08  7958 backup </tmp/x/delta/default/2013.08.18_09.49.21> copied to <Backup Copy>
INFO      2013.08.18 10:39:08  7958     moving backup to </tmp/x/delta/processedBackups/default>
INFO      2013.08.18 10:39:08  7958 backup </tmp/x/delta/default/2013.08.18_10.05.54> copied to <Backup Copy>
INFO      2013.08.18 10:39:08  7958     moving backup to </tmp/x/delta/processedBackups/default>
INFO      2013.08.18 10:39:08  7958 deleting in deltaCache </tmp/x/delta> processedBackups
INFO      2013.08.18 10:39:08  7958     age for deletion is > 99d (delete backups older than Sat 2013.05.11 10:39:08)
INFO      2013.08.18 10:39:08  7958 checking series <default>
INFO      2013.08.18 10:39:08  7958     default -> 2013.08.18_09.45.16 - not old enough to delete
INFO      2013.08.18 10:39:08  7958     default -> 2013.08.18_09.49.21 - not old enough to delete
INFO      2013.08.18 10:39:08  7958     default -> 2013.08.18_10.05.54 - not old enough to delete
STATISTIC 2013.08.18 10:39:08  7958                       duration = 1s
STATISTIC 2013.08.18 10:39:08  7958  [sec] |      user|    system
STATISTIC 2013.08.18 10:39:08  7958 -------+----------+----------
STATISTIC 2013.08.18 10:39:08  7958 process|      0.12|      0.01
STATISTIC 2013.08.18 10:39:08  7958 childs |      0.04|      0.00
STATISTIC 2013.08.18 10:39:08  7958 -------+----------+----------
STATISTIC 2013.08.18 10:39:08  7958 sum    |      0.16|      0.01 => 0.17
INFO      2013.08.18 10:39:08  7958 syncing ...
END       2013.08.18 10:39:08  7958 checking references and copying in <repl>

The following things have happened:

The missing backup was copied from delta cache (see log above):
```
copying </tmp/x/delta/default/2013.08.18_09.49.21> to <repl/default>
```
The last backup (2013.08.18_10.05.54) was not copied, because hat already had been done.
The program does some auto repair (automatically), which is normal for replications.

The program detected two uncompleted backups where the hard links (etc.) had to be set:

listing references:
  /tmp/x/repl/default/2013.08.18_09.49.21
    -> /tmp/x/repl/default/2013.08.18_09.45.16
  /tmp/x/repl/default/2013.08.18_10.05.54
    -> /x/repl/default/2013.08.18_09.49.21

Afterwards, it completed both backups.

Finally, I checked with storeBackupCheckBackup.pl if the replication is clean:

$ storeBackupCheckBackup.pl -c repl
BEGIN     2013.08.18 10:42:50  8042 checking backups in </tmp/x/repl>
VERSION   2013.08.18 10:42:50  8042 storeBackupCheckBackup.pl, 3.4 +
INFO      2013.08.18 10:42:50  8042 scanning directory <repl> for existing backups
INFO      2013.08.18 10:42:50  8042 scanning directory <repl/default> for existing backups
STATISTIC 2013.08.18 10:42:50  8042 found 1 backup series, 3 backups, 0 renamed backups
INFO      2013.08.18 10:42:50  8042 consistency check finished successfully
INFO      2013.08.18 10:42:50  8042 found no references to backups from lateLinks that need storeBackupUpdateBackup run
INFO      2013.08.18 10:42:50  8042 backup directories to check
INFO      2013.08.18 10:42:50  8042   /tmp/x/repl/default/2013.08.18_09.45.16
INFO      2013.08.18 10:42:50  8042   /tmp/x/repl/default/2013.08.18_09.49.21
INFO      2013.08.18 10:42:50  8042   /tmp/x/repl/default/2013.08.18_10.05.54
INFO      2013.08.18 10:42:50  8042 -- checking </tmp/x/repl/default/2013.08.18_09.45.16> ...
INFO      2013.08.18 10:42:50  8042 -- checking </tmp/x/repl/default/2013.08.18_09.49.21> ...
INFO      2013.08.18 10:42:50  8042 -- checking </tmp/x/repl/default/2013.08.18_10.05.54> ...
INFO      2013.08.18 10:42:50  8042 -- no WARNINGS OCCURRED DURING THE CHECK! --
INFO      2013.08.18 10:42:50  8042 -- no ERRORS OCCURRED DURING THE CHECK! --
END       2013.08.18 10:42:50  8042 checking backups in </tmp/x/repl>

Heinz-Josef Claes 2014-04-20