{"id":588,"date":"2022-07-14T19:05:34","date_gmt":"2022-07-14T19:05:34","guid":{"rendered":"https:\/\/terrabioappdev.wpenginepowered.com\/deciphering-a-mystery-workflow-written-in-wdl\/"},"modified":"2023-12-27T04:55:36","modified_gmt":"2023-12-27T04:55:36","slug":"deciphering-a-mystery-workflow-written-in-wdl","status":"publish","type":"post","link":"https:\/\/terra.bio\/deciphering-a-mystery-workflow-written-in-wdl\/","title":{"rendered":"Deciphering a mystery workflow written in WDL"},"content":{"rendered":"

Have you ever had to decipher a workflow written by someone else, and wished there was a quick way to figure out what it does and how? Earlier today, I gave a talk about this very topic at the 2022 Bioinformatics Open Source Conference (as part of <\/span>ISMB 2022<\/span><\/a>); and in the spirit of open-source, I’ve posted the <\/span>pre-recorded version on YouTube<\/span><\/a>!\u00a0<\/span><\/p>\n

In my talk, I presented a practical methodology for elucidating the structure and function of a workflow written in the Workflow Description Language, aka WDL. This systematic approach is intended to help bioinformaticians efficiently interpret and if necessary, reverse engineer existing WDL workflows. But first, a bit of context if you’re not already familiar with WDL.<\/span><\/p>\n

 <\/p>\n

What is WDL?<\/span><\/h3>\n

The Workflow Description Language (WDL, pronounced “widdle”) is a domain-specific language for describing data processing and analysis workflows. As such, it is not a full programming language; it is designed primarily as a means to define analysis tasks, chain them together in workflows, and parallelize their execution wherever possible. The language’s syntax strives to achieve portability across execution platforms, and lends itself especially well to execution on cloud infrastructure due to advanced support for containerization of individual steps. In addition, WDL is explicitly intended to be accessible to a wide range of users without requiring advanced programming experience.<\/span><\/p>\n

 <\/p>\n

\"\"<\/p>\n

Example WDL script that parallelizes variant calling across genomic intervals (see <\/span><\/i>blog post<\/span><\/i><\/a> for details)\u00a0<\/span><\/i><\/p>\n

 <\/p>\n

Originally created at the Broad Institute for developing GATK Best Practices workflows, the WDL language is now stewarded by the <\/span>OpenWDL<\/span><\/a> community, which was formed by individuals and organizations who adopted WDL in their own work.<\/span><\/p>\n

The main sources of documentation and educational resources for learning WDL are provided by <\/span>the OpenWDL organization on Github<\/span><\/a> (BSD-3). Most notably, the <\/span>WDL specification<\/span><\/a> describes the syntax and features of the language (variable types, built-in functions etc.), and the <\/span>learn-wdl<\/span><\/a> repository provides introductory materials (including <\/span>videos<\/span><\/a>) explaining basic syntax and demonstrating execution through hello-world level examples and some sample workflows.\u00a0<\/span><\/p>\n

 <\/p>\n

Reusable WDLs in the wild<\/span><\/h3>\n

Multiple consortia within the field of human genomics publish standardized workflows written in WDL, such as the Human Cell Atlas, and its adoption is spreading to non-human organisms and to related fields of study. For example, due to <\/span>recent events<\/span><\/a>, a growing number of public health laboratories in the US are adopting standardized WDL workflows for COVID-19 viral genomics and epidemiology.\u00a0<\/span><\/p>\n

Therefore it is increasingly likely for bioinformaticians to encounter workflows of interest written in WDL. Even if WDL is not their preferred language for developing their own workflows, bioinformaticians may need to be able to understand and apply these standard WDL workflows appropriately to their own data for continuity within a collaborative project, or they may want to reverse engineer and reimplement them in their preferred language.\u00a0<\/span><\/p>\n

 <\/p>\n

A systematic top-down approach to deciphering a random WDL<\/span><\/h3>\n

Real-world, full-scale workflows such as the GATK Best Practices typically involve more complex patterns than what is shown in the OpenWDL hello-world tutorials. Deciphering such workflows is a very different exercise compared to writing one’s own workflow from scratch, and can benefit from employing a systematic approach, rather than attempting to read through the code linearly.\u00a0<\/span><\/p>\n

One popular form for documenting realistic and\/or advanced use of a programming language is the “cookbook” style, in which the author posits a series of usage scenarios and shows how each would be implemented. This form is particularly helpful for developers who are interested in writing their own code. However, it is less helpful for someone who is trying to interpret an existing piece of code for reasons such as those outlined above.<\/span><\/p>\n

In my talk, I presented an alternate approach to satisfying the latter use case. First, I pretend to stumble across an existing real-world workflow that is written in WDL but is not sufficiently documented, then I proceed to deconstruct it systematically in order to understand what the workflow is meant to achieve and how it does it.<\/span><\/p>\n

The main take-home message here is the methodology itself, which can be adapted to other scenarios. As a secondary benefit, you will also gain some familiarity with WDL syntax and with interesting functional patterns of the language.<\/span><\/p>\n

 <\/p>\n