An important type of scientific workload is quite easy to parallelize, e.g.,

  • parameter exploration, i.e., running software on a data set with many different parameter settings;

  • running software on many different input files; or

  • a combination of the two scenarios above.

It is in fact so easy that it is called embarrassingly parallel. Since this workload is so common, we developed some frameworks to support them, and take the bookkeeping burden of your shoulders as much as possible: worker and atools.

Learning outcomes

When you complete this training you will be able to

  • to use the worker framework;

  • to use atools;

  • choose between both tools depending on the situations and your requirements;

  • understand weak versus strong parallel scaling;

  • recognize and avoid potential pitfalls.


Total duration: 3 hours.

Training materials

Slides are available in the GitHub repository, as well as example code and job scripts.

Repository and documentation of the tools covered:

Video materials

Video recordings of this training are available on YouTube.

  1. Introduction (1 minute)

  2. worker: parameter exploration (11 minutes)

  3. Get example scripts (1 minute)

  4. worker: Bash example (6 minutes)

  5. worker: MapReduce (2.5 minutes)

  6. worker features (17 minutes)

  7. parameter-weaver (5 minutes)

  8. worker: tuning (19 minutes)

  9. atools: parameter exploration (6.5 minutes)

  10. atools: features (7 minutes)

  11. atools: demo (6 minutes)

  12. atools: tuning (2 minutes)

  13. Comparison between worker and atools (2 minutes)

  14. File I/O and performance (4.5 minutes)

  15. Conclusions (1 minute)

  16. Implementation (3.5 minutes)

Target audience

This training is for you if you need to use HPC resources effectively for embarrassingly parallel workloads.


You will need to be comfortable using Linux and the HPC environment. If necessary, attend the appropriate training sessions on those subjects.


Geert Jan Bex (