[afnog] tar: Concatenate / Create from STDIN?

[Begumisa Gerald M]

Hi!

Thanks for responding.

      On Sun, 26 Feb 2006, Brian Candler wrote:
    > Writing a single file at a time into a tar is not likely to be the
    > most efficient way of working - but if by ./file1 you mean ./dir1
    > then that makes more sense. You can of course list multiple files
    > and/or directories on a single tar command line.

Yup.  Actually some are files and some are directories.  The reason why
only one file is being written at a time is because there are rather a
large number of files.

In fact, the above command is run from an AWK script which initially would
try to tar all files (and directories) and send them at a go (something
like tar -cf file1 file2 directory1 ...).  However the list got too big
that because of some AWK limitations (or maximum number of command line
arguments?), it would die before completing the operation so I thought I'd
send them one by one (in a loop).  What happens is that the input to the
AWK script is the output of the 'find' command which checks for all files
changed since a certain time.

    > [Aside: you might want to use '&&' instead of ';' to separate the
    > commands in this kind of scenario. This means that if one of the
    > intermediate commands fails, the later commands are not executed]

Cool.  Thanks.

    > If that doesn't work, you could try creating a named pipe (see
    > mkfifo) and appending that.
    >
    >
    >     # once only (this is permanent)
    >     mkfifo /home/me/myfifo
    >
    >     ...
    >
    >     cat >/home/me/myfifo & tar -Af /home/me/myfifo /backup/full-backup.tar

Thanks!  Looks neat!  I'll try this out.

    > I don't think so, because appending to an existing tarfile would
    > imply uncompressing it, editing it in-place as mentioned before, and
    > then recompressing the results. You can't edit a compressed file
    > in-place meaningfully.

Guess so, coming to think of it :-[

    > I suspect that there may be a better or simpler way to achieve what
    > you want to achieve (such as tarring all the required files in one
    > go at the source side), but that depends on the details of what
    > you're doing and why.

Probably there is.  As I mentioned basically what happens is I run 'find'
to get a list of all files changed in the last 'x' days and using an awk
script, tar these files, one by one (one by one because they got too many
to supply as an argument to tar) and send them over to the remote machine.

A possibility is to tar and zip these files to a temporary folder on the
local machine then send them over to the backup server, however, there's
an issue with the space that would be needed for this operation (I've got
only so much space), which is why I thought I'd pipe the output of tar to
ssh so that no additional disk space is used for this operation...

Any thoughts?


Cheers,
Gerald.

PS: The AWK script involved is a few lines, which is why am motivated to
write it, as opposed to installing bacula and friends :-)