[afnog] tar: Concatenate / Create from STDIN?

[Brian Candler]

On Sat, Feb 25, 2006 at 06:55:27PM +0300, Begumisa Gerald M wrote:
> Hi,
> 
> I'm faced with an interesting issue.  I'm using a simple command to backup
> a bunch of files from one machine to another.  I would like to end up with
> one .tar.gz file on the remote machine.
> 
> Basically, the command below is repeated for the entire list of files I'm
> backing up:
> 
> tar -cf - ./file1 | ssh backup-user at host "cat > /tmp/temp.tar; tar \
> -Af /backup/full-backup.tar /tmp/temp.tar; rm /tmp/temp.tar"

Writing a single file at a time into a tar is not likely to be the most
efficient way of working - but if by ./file1 you mean ./dir1 then that makes
more sense. You can of course list multiple files and/or directories on a
single tar command line.

> As you can see, on the remote machine, I create a temporary tar file then
> concatenate it with the full-backup.tar file which represents the full
> backup.  At the end of the operation, I gzip the full-backup.tar file.

[Aside: you might want to use '&&' instead of ';' to separate the commands
in this kind of scenario. This means that if one of the intermediate
commands fails, the later commands are not executed]

> My question is - is there a way to concatenate one tar file with another
> - where the second tar file comes from standard input.  This would help
> eliminate the extra step on the remote machine.  Something like:
> 
> tar -cf - ./file1 | ssh backup-user at host "tar -Af \
> /backup/full-backup.tar -"

It might work the other way round: -Af - /backup/full-backup.tar

The actual tarfile that you are concatenating onto must be a real file,
because I expect tar has to edit some of the existing data (e.g. seek to
end, wind back a bit, overwrite etc). The manpage says:

    Note: This option requires a rewritable tarfile

which implies it can't be stdin (since you can't both read and write
something which is on stdin)

If that doesn't work, you could try creating a named pipe (see mkfifo) and
appending that.

    # once only (this is permanent)
    mkfifo /home/me/myfifo

    ...

    cat >/home/me/myfifo & tar -Af /home/me/myfifo /backup/full-backup.tar

> Currently, it looks like the tar program will only read from standard
> input when extracting files and not when updating (creating / appending /
> concatenating) an archive.

OK, well I've not tested it myself.

> Secondly, might anyone know how to gzip everything, all in one step
> something like:
> 
> <wishful>
> tar -cf - ./file1 | ssh backup-user at host "tar -zAf \
> /backup/full-backup.tar.gz -"
> </wishful>

I don't think so, because appending to an existing tarfile would imply
uncompressing it, editing it in-place as mentioned before, and then
recompressing the results. You can't edit a compressed file in-place
meaningfully.

I suspect that there may be a better or simpler way to achieve what you want
to achieve (such as tarring all the required files in one go at the source
side), but that depends on the details of what you're doing and why.

Regards,

Brian.