[afnog] tar: Concatenate / Create from STDIN?

[Brian Candler]

On Mon, Feb 27, 2006 at 07:13:46PM +0300, Begumisa Gerald M wrote:
> In fact, the above command is run from an AWK script which initially would
> try to tar all files (and directories) and send them at a go (something
> like tar -cf file1 file2 directory1 ...).  However the list got too big
> that because of some AWK limitations (or maximum number of command line
> arguments?), it would die before completing the operation so I thought I'd
> send them one by one (in a loop).

Ah. Well, that might be a shell limitation (you could try /usr/local/bin/bash
instead of /bin/sh) or perhaps awk, depending on how
you're using it.

But there is a more elegant solution here: provide the list of
files/directories you want to backup on stdin, and then it's unlimited.

The traditional utility which works this way is cpio. With the appropriate
flags you can make it create output which is compatible with tar.

Read your own cpio manpage, but using FreeBSD 5.4 I can do something like:

cat <<EOF | cpio -o -H ustar | gzip -9 >foo.tar.gz
./file1
./file2
./file3
EOF

This also has the advantage that your filenames can contain spaces, because
each name is on its own line.

> What happens is that the input to the
> AWK script is the output of the 'find' command which checks for all files
> changed since a certain time.

cpio really is your friend here, as the output of find can be piped directly
to cpio (you can pre-filter it using awk or perl first if you wish).

Also, using the '-print0' flag to find, and the '-0' flag to cpio, your
filenames can even contain newlines :-) The list of filenames passed to cpio
is then separated with NUL characters.

Regards,

Brian.