GNU Astronomy Utilities manual

4.3.2 How to run simultaneous operations

There are two approaches to simultaneously execute a program: using GNU Parallel or Make (GNU Make is the most common implementation). The first is very useful when you only want to do one job multiple times and want to get back to your work without actually keeping the command you ran. The second is usually for (very) complicated processes, with lots of dependancies between the different products (for example a data-production pipeline).

GNU Parallel

When you only want to run multiple instances of a command on different threads and get on with the rest of your work, the best method is to use GNU parallel. Surprisingly GNU Parallel is one of the few GNU packages that has no Info documentation but only a Man page, see Info. So to see the documentation after installing it please run

$ man parallel

As an example, let’s assume we want to crop a region fixed on the pixels (500, 600) with the default width from all the FITS images in the ./data directory ending with sci.fits to the current directory. To do this, you can run:

$ parallel astimgcrop --numthreads=1 --xc=500 --yc=600 ::: \
  ./data/*sci.fits

GNU Parallel can help in many more conditions, this is one of the simplest, see the man page for lots of other examples. For absolute beginners: the backslash (\) is only a line breaker to fit nicely in the page. If you type the whole command in one line, you should remove it.

Make

Make is a utility built for specifying “targets”, “prerequisites” and “receipts”. It allows you to define very complicated dependancy structures for complicated processes that commonly start off with a large list of inputs and builds them based on the dependancies you define. GNU Make³⁸ is the most common implementation which (like nearly all GNU programs comes with a wonderful manual³⁹). It is very basic and short (the most important part is about 100 pages).

Make has the --jobs (-j) which allows you to specify the maximum number of jobs that can be done simultaneously. For example a common 4 physical core CPU usually has 8 processing threads. So you can run:

$ make -j8

Once the dependancy tree for your processes is built, Make will run the independent targets simultaneously.

Read in other formats.
GNU Astronomy Utilities manual, November 2015.