btar - Block TAR
Tar-compatible archiver which allows arbitrary compression and ciphering, redundancy, differential backup, indexed extraction, multicore compression, input and output serialisation, and tolerance to partial archive errors.
- Single file backup
- Optional indexed access
- Serialisable listing, backup and restore
- Backups (btar files) can be restored with usual unix tools (sh and tar)
- Allows any compression, ciphering, etc. (if you have a program that does so), and still resist partial corruption (-F)
- Can add an erasure code (block XOR of the others) that can protect against one erasure (-R)
- Can do quick incremental backups (rearchive only the files that changed) (-d/-D)
- Can work over a GNU tar output (even indexing it)
- Can be used in pipelines, with little memory usage even for big backup sizes
- Can parallelize the filter operations on any number of processes (compression, ciphering, ...) (-j)
- Can extract parts of the backup filtering (uncompressing, deciphering) only the relevant blocks
- Sending USR1 will show a progress report on stderr (on btar file creation)
- If linked with librsync, will only save deltas of big files on differential backup (-Y)
- The files can be re-filtered (-m) through pipes, not requiring extraction and reachiving.
- 1.1.1 is simply in a "works for the me (author)" state. I store my backups *only* with it.
- The incremental backups are only based on mtime.
- Only supports regular files, directories and symlinks. No hard link storage
- A small tool provided (xortest) can recover one full lost block from the btar. But a better tool could be written, for non overlapping in-block small erasures. Maybe based on a ddrescue log file.
Trunk version: trunk tarball (should be stable, and possibly with fixes after the release)
Backup the home with xz compression with 4 cores:
# btar -F xz -j 4 -f /mnt/usbb/backup/myhomebackup.btar -c /home/
Update to a diff backup:
# btar -F xz -j 4 -d /mnt/usbb/backup/myhomebackup.btar \ -f /mnt/usbb/backup/myhomebackup_diff.btar \ -c /home/
Backup the mail using 4 cores with gpg, and add an XOR redundancy. Use gpg-agent or it will ask a password per block.
$ btar -F "gpg2 -se" -j 4 -R -c mail > /mnt/usbb/backup/mymailbackup.btar
Extract only my mail from the home backup made with xz (or whatever it was):
$ btar -x -f /mnt/usbb/backup/myhomebackup.btar 'home/viric/mail/*'
Test that a btar file contents are still the same on disk (although if a file is missing in the btar, it will not report it):
$ btar -T -f /mnt/usbb/backup/myhomebackup.btar | tar d
btar 1.1 [e5b0e4f692] Copyright (C) 2011-2012 Lluis Batlle i Rossell usage: btar [options] [actions] actions: -c Create a btar file. Non-options mean directories or files to add. -x Extract the btar contents to disk. In this case, non-options mean glob patterns to extract. -T Extract the btar contents as a tar to stdout. In this case, non-options mean glob patterns to extract. -l List the btar index contents. -L Output the btar index as tar. -m Mangle filters and block size from stdin to output btar (-f or stdout)). (none) Make btar file from the standard input data (filter mode). options only meaningful when creating or filtering: -b <blocksize> Set the block size in megabytes (default 10MiB) -d <file> Take the index in the btar file as files already stored -D <file> Take the index file as files already stored -f <file> Output file while creating, input while extracting, or stdin/out if ommitted. -F <filter> Filter each block through program named 'filter'. -H Delete files as noted in diff backups, when extracting. -j <n> Number of blocks to filter in parallel. -N Skip making an index in the btar, make only blocks. -R Add a XOR redundancy block. -U <filter> Filters for the index and deleted list. -v Output the file names on stderr (on action 'c'). -X <pattern> Add glob exclude pattern. -Y Create and use rsync signatures in indices for binary diff. other options: -G <defilter> Defilter each input block through program named 'filter'. May be relevant for '-d' when creating/filtering, or extracting. -V Show traces of what goes on. More V mean more traces. examples: tar c /home | btar -b 50 -F xz > /tmp/homebackup.btar btar -F xz -c -f mydir.btar mydir btar -d mydir.btar -v -c mydir > mydir2.btar btar -T 'mydir/m*' < mydir.btar | tar x This program comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. See the source distribution for details.
Licence: GPLv3 or later
Author: Lluís Batlle i Rossell <email@example.com> (without the word tar)