DRAGEN is a platform provided by Illumina that offers accurate, comprehensive, and efficient secondary analysis of next-generation sequencing (NGS) data with a significant speed-up over tools that are commonly used for such tasks.
The improved performance offered by DRAGEN is possible due to the use of Illumina proprietary algorithms in conjunction with a special type of hardware accelerator called field programmable gate arrays (FPGAs). For example, when using AWS, FPGAs are available via the F1 instance type.
Running DRAGEN on Nextflow Tower#
We have extended the Tower Forge feature for AWS Batch to support DRAGEN. Tower Forge ensures that all of the appropriate components and settings are automatically provisioned when creating a Compute Environment for executing pipelines.
When deploying data analysis workflows, some tasks will need to use normal instance types (e.g. for non-DRAGEN processing of samples) and others will need to be executed on F1 instances. If the DRAGEN feature is enabled, Tower Forge will create an additional AWS Batch compute queue which only uses F1 instances, to which DRAGEN tasks will be dispatched.
To showcase the capability of this integration, we have implemented a proof of concept pipeline called nf-dragen. To run it, sign-in into Tower, navigate to the Community Showcase and select the “nf-dragen” pipeline.
You can run this pipeline at your convenience without any extra setup. Note however that it will be deployed in the Compute Environment owned by the Community Showcase.
To deploy the pipeline on your own AWS cloud infrastructure, please follow the instructions in the next section.
Deploy DRAGEN in your own workspace#
DRAGEN is a commercial technology provided by Illumina, so you will need to purchase a license from them. To run on Tower, you will need to obtain the following information from Illumina:
- DRAGEN AWS private AMI ID
- DRAGEN license username
- DRAGEN license password
Tower Forge automates most of the tasks required to set up an AWS Batch Compute Environment. Please follow our guide for more details.
In order to enable support for DRAGEN acceleration, simply toggle the “Enable DRAGEN” option when setting up the Compute Environment via Tower Forge.
In the “DRAGEN AMI Id” field, enter the AWS AMI ID provided to you by Illumina.
Please ensure that the Region you select contains DRAGEN F1 instances.
Pipeline implementation & deployment#
label directive allows you to annotate a process with mnemonic identifiers of your choice. Tower will use the
dragen label to determine which processes need to be executed on DRAGEN F1 instances.
1 2 3 4 5
Please refer to the Nextflow label docs for more information.
- Define Secrets
At Seqera, we use Secrets to safely encrypt sensitive information when running licensed software via Nextflow. This enables our team to use the DRAGEN software safely via the
nf-dragen pipeline without having to worry about the setup or safe configuration of the license key. These Secrets will be provided securely to the
--lic-server option when running DRAGEN on the CLI to validate the license.
In the nf-dragen pipeline, we have defined two Secrets called
DRAGEN_PASSWORD, which you can add via the Tower UI by going to “Secrets -> Add Pipeline Secret”:
Please refer to the Secrets documentation for more information about this feature.
DRAGEN integration with Tower is currently only available for use on AWS, however, we plan to extend the functionality to other supported platforms like Azure in the future.