Tutorial 4 & 5: Workflow Management
Time: May 11-12 & June 29-30
Day 1: Atomic Simulation Recipes for automated computational workflows
This lecture will introduce the Atomic Simulation Recipes (ASR) – a new open source Python framework for working with atomistic materials simulations in an efficient and sustainable way that is ideally suited for high-throughput studies. Central to ASR is the concept of a Recipe: a high-level Python script that performs a well-defined simulation task robustly and accurately while automatically keeping track of the data provenance. Being independent objects with automatic data provenance logging, Recipes may be freely combined through Python scripting giving maximal freedom for users to build advanced workflows that can be executed using workflow managers like the MyQueue task scheduler[1]. The ASR leverages the functionalities of the Atomic Simulation Environment (ASE)[2] to interface with external simulation codes, attain a high abstraction level, and create databases that may be easily presented as web pages.
After the introduction, we offer tutorials that illustrate the main ASR functionalities including: how to write your own Recipe, how to construct a Python workflow by combining different Recipes, how to collect the results into an ASR database, and how to generate webpages to present the database in a browser.
References
[1] MyQueue: Task and workflow scheduling system, Mortensen et al., 2020 Journal of Open Source Software, 5(45), 1844, https://doi.org/10.21105/joss.01844
[2] The atomic simulation environment—a Python library for working with atoms, Larsen et al., 2017 J. Phys.: Condens. Matter, 29, 273002, https://doi.org/10.1088/1361-648X/aa680e
Schedule
All times in CEST:
10:00-10:45 Introduction to ASE and ASR (by Kristian Thygesen)
11:00-13:00 Demonstration of ASE and ASR (by Ask Larsen)
Recordings of Part 1 (May 11, 2021)
Introduction to ASE and ASR by Kristian Thygesen:
Live demonstration of ASE and ASR by Ask Larsen:
Day 2: Workflow management and execution with FireWorks
We are happy to announce the next tutorial on the FireWorks [1] workflow manager. FireWorks is a free, open-source code for defining, managing, and executing workflows. Workflow execution can be automated over arbitrary computing resources, including those that have a queueing system. FireWorks has been used to run millions of workflows encompassing tens of millions of CPU-hours across diverse application areas and in long-term production projects over the span of multiple years. As an example, FireWorks has been the tool of choice for the Materials Project [2], which is an open-access database of hundreds of thousands of computed properties (ground-state, electronic band structures, phonon band structures, elastic and piezoelectric tensors, ...) for inorganic compounds, nanoporous materials and molecules.
The tutorial will be focused on the usage of FireWorks, with a series of hands-on to practice the definition and execution of workflows. We will start with an overview of the main concepts of FireWorks, in particular the centralized database model which is at the core of FireWorks. The participants will then have a series of hands-ons to directly try out FireWorks and its capabilities. A small recap of the concepts covered will be provided at the end of each hands-on.
List of hands-ons:
- Quickstart tutorial
- Running workflows through a queue manager, e.g. PBS or SLURM
- Introduction to FireWorks
- Defining jobs using "Firetasks"
- Creating Workflows
An introduction to atomate [3], a library of ready-to-use workflows dedicated to materials science, will also be presented at the end of the tutorial.
[1] https://materialsproject.github.io/fireworks/, A. Jain et al., FireWorks: a dynamic workflow system designed for high-throughput applications. Concurrency Computat.: Pract. Exper., 27, 5037 (2015). [2] https://materialsproject.org/, A. Jain et al., The Materials Project: A materials genome approach to accelerating materials innovation, APL Materials 1, 011002 (2013). [3] https://atomate.org/, M. K. Montoya et al., Atomate: A high-level interface to generate, execute, and analyze computational materials science workflows, Comput. Mater. Sci. 139,140 (2017).
Schedule
All times in CEST:
10:00-10:45 Introduction to FireWorks (by David Waroquiers)
11:00-13:00 Live demonstration of basic functionalities of FireWorks (by David Waroquiers and Guido Petretto)
Recordings of Part 1 (May 12, 2021)
Introduction to FireWorks by David Waroquiers:
Live demonstration of basic functionalities of FireWorks by David Waroquiers and Guido Petretto: