If you have a bunch of smaller jobs you have to go trough you might think of parallelization. While some software offers native parallel execution many do not. Python offers a very simple way to parallelize OpenMP jobs (no shared memory). An example.
Well, for sure the example above is relatively useless as the worker
method only returns a print of the inputs it gots. However, it demonstrates
the OpenMP parallelization procedure. For a more practical use the worker
should do something useful. I wrote this example as I got asked by a friend
whether it would be possible to start a visualization process (using GrADS)
GrADS is mainly used to visualize weather or climate forecasts or atmospheric
analysis. GrADS uses the grads visualization language—or in other words—small
.gs script files as we have defined them within the par list in the
snippet above. Within the worker you can actually do whatever you need. In
this example we would like to execute a (linux) binary grads given some
To execute system-binaries the system call can be used within python.
Therefore, the worker has to be tuned (in contrast to the very simple
worker shown above).
Please note that this is not the way to go. Especially as you do not
have any control of possible outputs or errors procuded by the system
calls. An alternative is using pythons subprocess library.
proper error handling
stdout and stderr can be fetched AND analyzed/processed if needed.
If required you can directly fetch the output of the script and process
it in python. Or you analize the error output and take further action
such as e.g., sending error log e-mails, restart the process, start
a fallback procedure, …
subprocess.Popen takes the command in a list format. An example:
if you would call ls -l in the call would be subprocess.Popen(['ls','-l'],...).
In this example we would like to call something like grads -blx -c run t2m.gs: