Ttdfjt

Complications

The following will work only sometimes:

gzip -d `cat large_file_list`

Three problems are (in bash and most other Bourne-like shells):

1. It will fail if any file name has space tab or newline characters in it (assuming $IFS has not been modified). This is because of the shell's word splitting.




2. It is also liable to fail if any file name has glob-active characters in it. This is because the shell will apply pathname expansion to the file list.




3. It will also fail if filenames starts with - (if POSIXLY_CORRECT=1 that only applies to the first file) or if any filename is -.




4. It will also fail if there are too many file names in it to fit on one command line.

The code below is subject to the same problems as the code above (except for the fourth)

for file in `cat large_file_list`
do
 gzip -d $file
done

Reliable solution

If your large_file_list has exactly one file name per line, and a file called - is not among them, and you're on a GNU system, then use:

xargs -rd'n' gzip -d -- <large_file_list

-d'n' tells xargs to treat each line of input as a separate file name.

-r tells xargs not to run the command if the input file is empty.

-- tells gzip that the following arguments are not to be treated as options even if they start with -. - alone would still be treated as - instead of the file called - though.

xargs will put many file names on each command line but not so many that it exceeds the command line limit. This reduces the number of times that a gzip process must be started and therefore makes this fast. It is also safe: the file names will also be protected from word splitting and pathname expansion.

I doubt it would matter much.

I would use a loop, just because I don't know how many files are listed in the list file, and I don't (generally) know if any of the filenames have spaces in their names. Doing a command substitution that would generate a very long list of argument may result in an "Argument list too long" error when the length of the list generated is too long.

My loop would look like

while IFS= read -r name; do
 gunzip "$name"
done <file.list

This would additionally allow me to insert commands for processing the data after the gunzip command. In fact, depending on what the data actually is and what needs to be done with it, it may even be possible to process it without saving it to file at all:

while IFS= read -r name; do
 zcat "$name" | process_data
done <file.list

(where process_data is some pipeline that reads the uncompressed data from standard input)

If the processing of the data takes longer than the uncompressing of it, the question of whether a loop is more efficient or not becomes irrelevant.

Ideally, I would prefer to not work off a list of filenames though, and instead use a filename globbing pattern, as in

for name in ./*.gz; do
 # processing of "$name" here
done

where ./*.gz is some pattern that matches the relevant files. This way we are not depending on the number of files nor on the characters used in the filenames (they may contain newlines or other whitespace characters, or start with dashes, etc.)

Related:

• Understanding "IFS= read -r line"

Out of those two, the one with all files passed to a single invocation of gzip is likely to be faster, exactly because you only need to launch gzip once. (That is, if the command works at all, see the other answers for the caveats.)

But, I'd like to remind of the golden rule of optimization: Don't do it prematurely.

1. 
   

   Don't optimize that sort of thing before you know it's a problem.

   
   
   

   Does this part of the program take a long time? Well, decompressing large files might, and you're going to have to do it anyway, so it might not be that easy to answer.

   
   


2. 
   

   Measure. Really, it's the best way to be sure.

   
   
   

   You'll see the results with your own eyes (or with your own stopwatch), and they'll apply to your situation which random answers on the Internet might not. Put both variants in scripts and run time script1.sh, and time script2.sh. (Do that with a list of empty compressed files to measure the absolute amount of the overhead.)

   
   

How fast is your disk?

This should use all your CPUs:

parallel -X gzip -d :::: large_file_list

So your limit is likely going to be the speed of your disk.

You can try adjusting with -j:

parallel -j50% -X gzip -d :::: large_file_list

This will run half of the jobs in parallel as the previous command, and will stress your disk less, so depending on your disk this can be faster.

ADDED after some profiling:

I made one hundred dirs 00 to 99 on a tmpfs and filled them with man-1 files starting with "s". That gives me 113 files from 20 byte to 69 K in each dir, totalling to 1.2 Mb in each dir.

Result: it ALL works:

Globbing and huge arg list:

gzip -d ??/* 
and
gzip -d $(<filelist)

both 0.8 s

With a loop:

for file in $(<filelist)
do
 gzip -d $file 
done

14 sec

And when I add my ampersand:

... 
 gzip -d $file &
... 

5.8 sec

The performance side of this is easily explained: the less you call gzip and the less you wait for it, the faster it runs.

Now: Is there at all a maximum arg list length? Is there a maximum number of processes? (I have ulimit -u of 31366, but it looks like I could even raise that, as root on my 300$ i5 Mini-PC...). Or will the max. be "only limited by the amount of avail. Memory".

Despite the overhead of ampersanding (forking off) 11300 mostly tiny jobs, this takes half as long as to wait for each command to complete. And this was on a tmpfs...

I forgot this one:

time find ?? -name "*.gz" -exec gzip -d +

also 0.8 s (6.2 s without the "+", but with ugly ";")

ORIGINAL answer:

Short answer: use ampersand & to execute each gzip command in the background to start the next loop iteration immediately.

for filnam in $(<flist)
do
 gzip -d $filnam &
done

I also replaced cat with "<" (not in a loop, so it is not critical) and backticks with $() (more "modern").

to test/benchmark: place the shell builtin time in front of your command: I did time . loop.cmd (with loop.cmd as shown above). This gives you the time spent, and after hitting Enter again the shell shows you the finished gzip "jobs".

I tested this out some weeks ago with a (small) list of (medium size) tar files to extract. The & made such a difference in time I could hardly believe it. The & has a double effect: it not only starts the next gzip immediately, it also gives you the prompt as soon as the last gzip got started.

CAREFUL: if you have whitespace in your filenames, it doesn't work. I don't know how to implement that. I tried with quotes and IFS...

Another well-known tip is: use globbing, if possible. Maybe start with a list of directories and for each go (cd $dir; gzip -d *.gz &).

--> I only can give you this ampersand "trick". First step is always optimizing the design. I am not sure what exactly you want this huge list of files for? Is it a hand made list? If it is generated by rules, then maybe use these rules directly and not via a filelist.

Your second example should be a bit faster, because gzip starts only once. But: I don't know what the limit is if you go gzip -d 1.gz 2.gz 3.gz ... 9999.gz, i.e. giving gzip a very large argument list.

Whitespace: a filename like '10 A.gz' results in gzip -d ... 9.gz 10 A.gz 11.gz.... Gzip will say: '10' and 'A.gz' not found...unless these files DO exist, and if it is rm and not gzip then you have that classic shell trap.

added:

What about:

find $(<dirlist) -name "*.gz" -exec gzip -d +

...starting from a smaller list of dirs? The + is the "parallelization" option (a find -exec thing, like ...).

If you can redesign a bit, this must be the best...