Azure Batch

This guide assumes you have an existing Azure account.

Depending on your region and subscription type, a newly-created account may not be entitled to any VMs without first making a service request to Azure. See Azure Batch service quotas and limits for more information.

There are two ways to create an Azure Batch compute environment in Seqera Platform:

• Tower Forge: This option automatically creates the Azure Batch resources needed for your Seqera compute environment. This eliminates the need to set up your Azure Batch infrastructure manually.

• Manual: This option allows Seqera to use existing Azure Batch resources.

Tower Forge

Tower Forge automates the configuration of an Azure Batch compute environment and the resources required for deploying Nextflow pipelines. This is recommended if you do not yet have an Azure Batch environment fully set up.

Tower Forge creates resources that may have associated costs in your Azure account.

Resource group

To create Azure Batch and Azure Storage accounts, first create a resource group in the region of your choice.

Create a resource group

1. Log in to your Azure account and select Create new resource group on this page.

2. Enter a name for the resource group, e.g., towerrg.

3. Select the preferred region for this resource group.

4. Select Review and Create to proceed to the review screen.

5. Select Create.

Storage account

After you have create a resource group, set up an Azure storage account.

Create a storage account

1. Log in to your Azure account and select Create a storage account on this page.

2. Enter a name for the storage account, e.g., towerrgstorage.

3. Select the preferred region for this storage account.

4. Select Review and Create to proceed to the review screen.

5. Select Create to create the Azure Storage account.

6. Go to your new storage account and select Container.

7. Select + Container to create a new Blob storage container.

   A new container dialogue will open. Enter a suitable name, e.g., towerrgstorage-container.

8. Go to the Access Keys section of your new storage account (towerrgstorage in this example).

9. Store the access keys for your Azure Storage account, to be used when you create a Seqera compute environment.

   Blob container storage credentials are associated with the Batch pool configuration when it is created. Once your compute environment has been created with Tower Forge, these credentials should not be changed in your Seqera instance.

Batch account

After you have created a resource group and storage account, create a Batch account.

Create a Batch account

1. Log in to your Azure account and select Create a batch account on this page.

2. Enter a name for the Batch account, e.g., towerrgbatch.

3. Select the preferred region for this Batch account.

4. Select Review and Create to proceed to the review screen.

5. Select Create.

Compute environment

Tower Forge automates the configuration of an Azure Batch compute environment and the queues required for the deployment of Nextflow pipelines.

After your Azure resources are set up, create a new compute environment in Seqera.

Create a Tower Forge Azure Batch compute environment

1. In a workspace, select Compute Environments > New Environment.

2. Enter a descriptive name for this environment, e.g., Azure Batch (east-us).

3. Select Azure Batch as the target platform.

4. From the Credentials drop-down, select existing Azure credentials, or select + to add new credentials. If you choose to use existing credentials, skip to step 7.

   You can create multiple credentials in your Seqera environment. See Credentials.

5. Enter a name, e.g. Azure Credentials.

6. Add the Batch account and Blob Storage credentials that we created previously.

7. Select a Region, e.g., eastus (East US).

8. In the Pipeline work directory field, enter the Azure blob container created previously, e.g., az://towerrgstorage-container/work.

   The blob container must be in the same Region from step 7.

9. Set the Config mode to Batch Forge.

10. Enter the default VM type, depending on your quota limits. The default is Standard_D4_v3.

11. Enter the VMs count. This is the number of VMs you wish to deploy.

12. Enable Autoscale to scale up and down automatically, based on the number of pipeline tasks. The number of VMs will vary from 0 to VMs count.

13. Enable Dispose resources for Seqera to automatically delete the Batch pool once the workflow is complete.

14. Select or create Container registry credentials to authenticate to an Azure registry (used by the Wave containers service).

15. Apply Resource labels to the cloud resources consumed by this compute environment. Workspace default resource labels are prefilled.

16. Expand Staging options to include optional pre- or post-run Bash scripts that execute before or after the Nextflow pipeline execution in your environment.

17. Specify custom Environment variables for the Head job and/or Compute jobs.

18. Configure any advanced options you need:

    • Use Jobs cleanup policy to control how jobs are deleted upon workflow completion.

    • Use Token duration to control the duration of the SAS token generated by Nextflow.

19. Select Create to finalize the compute environment setup. It will take a few seconds for all the resources to be created before the compute environment is ready to launch pipelines.

See Launch pipelines to start executing workflows in your Azure Batch compute environment.

Manual

This section is for users with a pre-configured Azure environment. This requires an existing Azure Batch account and Storage account.

Create a manual Seqera Azure Batch compute environment

1. In a workspace, select Compute Environments > New Environment.

2. Enter a descriptive name for this environment, e.g., Azure Batch (east-us).

3. Select Azure Batch as the target platform.

4. Select your existing Azure credentials or select + to add new credentials. If you choose to use existing credentials, skip to step 7.

   You can create multiple credentials in your Seqera environment. See Credentials.

5. Enter a name, e.g., Azure Credentials.

6. Add the Batch account and Blob Storage credentials that we created previously.

7. Select a Region, e.g., eastus (East US).

8. In the Pipeline work directory field, add the Azure blob container created previously, e.g., az://towerrgstorage-container/work.

   The Blob container must be in the same Region specified in step 7 above.

9. Set the Config mode to Manual.

10. Enter the Compute Pool name. This is the name of the Azure Batch pool provided by your Azure administrator.

11. Specify custom Environment variables for the Head job and/or Compute jobs.

12. Configure any advanced options you need:

    • Use Jobs cleanup policy to control how jobs are deleted upon workflow completion.
    • Use Token duration to control the duration of the SAS token generated by Nextflow.

13. Select Create to finalize the compute environment setup. It will take a few seconds for all the resources to be created before you are ready to launch pipelines.

See Launch pipelines to start executing workflows in your Azure Batch compute environment.